2024-09-11T04:54:36.339 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T04:54:36.357 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T04:54:36.444 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187 branch: wip-hemanth-testing-2024-09-10-0723 description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} email: hyelloji@redhat.com first_in_suite: false job_id: '7900187' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-hemanth-testing-2024-09-10-0723 ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-hemanth-testing-2024-09-10-0723 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 owner: scheduled_hyelloji@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9167 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-hemanth-testing-2024-09-10-0723 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 targets: smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEm+gnqRdQvsuNDGHweFWWmcOD141W3VrgD2CVsm9QGIKs7gItQxwoWGtSYdJw1uFI+7KGxvsycXI8MKIbIxzyE= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2pNAXdrQqddEiRETthl1ihy2Tl+R5DLs9FNtzsrdOXAA7B9hNZF5TKylK4IzTnLzmuNyT4d/Sxbgp3Dk/ebrI= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:16:14 tube: smithi user: hyelloji verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T04:54:36.444 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa; will attempt to use it 2024-09-11T04:54:36.445 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks 2024-09-11T04:54:36.445 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T04:54:36.446 INFO:teuthology.task.internal:Checking packages... 2024-09-11T04:54:36.469 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8f219c476cb4f26fc5719c661c0323719a45cb30' 2024-09-11T04:54:36.469 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T04:54:36.469 INFO:teuthology.packaging:ref: None 2024-09-11T04:54:36.469 INFO:teuthology.packaging:tag: None 2024-09-11T04:54:36.469 INFO:teuthology.packaging:branch: wip-hemanth-testing-2024-09-10-0723 2024-09-11T04:54:36.469 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T04:54:36.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-hemanth-testing-2024-09-10-0723 2024-09-11T04:54:36.617 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4854-g8f219c47-1jammy 2024-09-11T04:54:36.619 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T04:54:36.625 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T04:54:36.625 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T04:54:36.693 INFO:teuthology.task.internal:Saving configuration 2024-09-11T04:54:36.707 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T04:54:36.728 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T04:54:36.752 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 04:49:11.867366', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINsCDh8r+XIGhwRSUBQhd9VuYyersLedD4S7nL0LwIL7'} 2024-09-11T04:54:36.775 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 04:49:11.865931', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM5g7hvLTQuvxMJRm01XLIupZ7FVCjlIy2fpkWCKxjCq'} 2024-09-11T04:54:36.776 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T04:54:36.781 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-11T04:54:36.781 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-11T04:54:36.781 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T04:54:36.876 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f755b607be0>, signals=[15]) 2024-09-11T04:54:36.876 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T04:54:36.882 INFO:teuthology.task.internal:Opening connections... 2024-09-11T04:54:36.882 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T04:54:36.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:54:37.006 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T04:54:37.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:54:37.083 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T04:54:37.089 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-09-11T04:54:37.094 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-09-11T04:54:37.094 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-09-11T04:54:37.142 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Ubuntu" 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="22.04" 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_CODENAME=jammy 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:ID=ubuntu 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE=debian 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T04:54:37.143 INFO:teuthology.orchestra.run.smithi138.stdout:UBUNTU_CODENAME=jammy 2024-09-11T04:54:37.144 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-09-11T04:54:37.170 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-09-11T04:54:37.174 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-09-11T04:54:37.174 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-09-11T04:54:37.219 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T04:54:37.219 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Ubuntu" 2024-09-11T04:54:37.219 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="22.04" 2024-09-11T04:54:37.219 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T04:54:37.219 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_CODENAME=jammy 2024-09-11T04:54:37.219 INFO:teuthology.orchestra.run.smithi190.stdout:ID=ubuntu 2024-09-11T04:54:37.220 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE=debian 2024-09-11T04:54:37.220 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T04:54:37.220 INFO:teuthology.orchestra.run.smithi190.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T04:54:37.220 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T04:54:37.220 INFO:teuthology.orchestra.run.smithi190.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T04:54:37.220 INFO:teuthology.orchestra.run.smithi190.stdout:UBUNTU_CODENAME=jammy 2024-09-11T04:54:37.220 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-09-11T04:54:37.244 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T04:54:37.253 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T04:54:37.293 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T04:54:37.293 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-09-11T04:54:37.296 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-09-11T04:54:37.301 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T04:54:37.326 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T04:54:37.326 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-09-11T04:54:37.344 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-09-11T04:54:37.351 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T04:54:37.429 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T04:54:37.443 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T04:54:37.443 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T04:54:37.444 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T04:54:37.444 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T04:54:37.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:54:37.449 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-09-11T04:54:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T04:54:37.450 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-11T04:54:37.495 INFO:teuthology.orchestra.run.smithi138.stdout:5.15.0-56-generic 2024-09-11T04:54:37.495 INFO:teuthology.task.kernel:Running kernel on smithi138: 5.15.0-56-generic 2024-09-11T04:54:37.495 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get clean 2024-09-11T04:54:37.496 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-56-generic 2024-09-11T04:54:37.497 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.15.0-56-generic 2024-09-11T04:54:37.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2024-09-11T04:54:37.617 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2024-09-11T04:54:37.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-09-11T04:54:37.736 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T04:54:37.754 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T04:54:37.818 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T04:54:37.855 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T04:54:37.915 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T04:54:37.915 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T04:54:37.953 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T04:54:37.968 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T04:54:38.026 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T04:54:38.063 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T04:54:38.065 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T04:54:38.066 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T04:54:38.066 INFO:teuthology.orchestra.run.smithi138.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T04:54:38.081 INFO:teuthology.orchestra.run.smithi138.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T04:54:38.084 INFO:teuthology.orchestra.run.smithi138.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T04:54:38.084 INFO:teuthology.orchestra.run.smithi138.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T04:54:38.089 INFO:teuthology.orchestra.run.smithi138.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T04:54:38.099 INFO:teuthology.orchestra.run.smithi138.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T04:54:38.101 INFO:teuthology.orchestra.run.smithi138.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T04:54:38.101 INFO:teuthology.orchestra.run.smithi138.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T04:54:38.101 INFO:teuthology.orchestra.run.smithi138.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T04:54:38.119 INFO:teuthology.orchestra.run.smithi138.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T04:54:38.119 INFO:teuthology.orchestra.run.smithi138.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T04:54:38.119 INFO:teuthology.orchestra.run.smithi138.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T04:54:38.120 INFO:teuthology.orchestra.run.smithi138.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T04:54:38.120 INFO:teuthology.orchestra.run.smithi138.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T04:54:38.133 INFO:teuthology.orchestra.run.smithi138.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T04:54:38.133 INFO:teuthology.orchestra.run.smithi138.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T04:54:38.150 INFO:teuthology.orchestra.run.smithi138.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T04:54:38.150 INFO:teuthology.orchestra.run.smithi138.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T04:54:38.150 INFO:teuthology.orchestra.run.smithi138.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T04:54:38.291 INFO:teuthology.orchestra.run.smithi138.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T04:54:38.324 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T04:54:38.342 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T04:54:38.463 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T04:54:38.574 INFO:teuthology.orchestra.run.smithi138.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T04:54:38.633 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T04:54:38.658 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T04:54:38.745 INFO:teuthology.orchestra.run.smithi138.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T04:54:38.758 INFO:teuthology.orchestra.run.smithi138.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T04:54:38.758 INFO:teuthology.orchestra.run.smithi138.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T04:54:38.806 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T04:54:38.816 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T04:54:38.818 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T04:54:38.819 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T04:54:38.829 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T04:54:38.830 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T04:54:38.832 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T04:54:38.844 INFO:teuthology.orchestra.run.smithi138.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T04:54:38.845 INFO:teuthology.orchestra.run.smithi138.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T04:54:38.864 INFO:teuthology.orchestra.run.smithi138.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T04:54:38.864 INFO:teuthology.orchestra.run.smithi138.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T04:54:38.901 INFO:teuthology.orchestra.run.smithi138.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T04:54:38.901 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T04:54:38.917 INFO:teuthology.orchestra.run.smithi138.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T04:54:38.921 INFO:teuthology.orchestra.run.smithi138.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T04:54:38.922 INFO:teuthology.orchestra.run.smithi138.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T04:54:38.922 INFO:teuthology.orchestra.run.smithi138.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T04:54:38.950 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T04:54:38.950 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T04:54:38.950 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T04:54:38.951 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T04:54:38.962 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T04:54:38.962 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T04:54:38.968 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T04:54:38.978 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T04:54:38.990 INFO:teuthology.orchestra.run.smithi138.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T04:54:38.990 INFO:teuthology.orchestra.run.smithi138.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T04:54:38.994 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T04:54:38.997 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T04:54:39.000 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T04:54:39.000 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T04:54:39.001 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T04:54:39.001 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T04:54:39.001 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T04:54:39.001 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T04:54:39.001 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T04:54:39.051 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T04:54:39.052 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T04:54:39.063 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T04:54:39.063 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T04:54:39.064 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T04:54:39.065 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T04:54:39.145 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T04:54:39.145 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T04:54:39.146 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T04:54:39.146 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T04:54:39.146 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T04:54:44.418 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 15.8 MB in 2s (7,570 kB/s) 2024-09-11T04:54:45.251 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 15.8 MB in 3s (5,826 kB/s) 2024-09-11T04:54:45.899 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-09-11T04:54:45.918 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T04:54:45.981 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-09-11T04:54:46.190 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2024-09-11T04:54:46.191 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2024-09-11T04:54:46.381 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T04:54:46.381 INFO:teuthology.orchestra.run.smithi138.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T04:54:46.381 INFO:teuthology.orchestra.run.smithi138.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T04:54:46.382 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T04:54:46.382 INFO:teuthology.orchestra.run.smithi138.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T04:54:46.382 INFO:teuthology.orchestra.run.smithi138.stdout: python2.7-minimal 2024-09-11T04:54:46.382 INFO:teuthology.orchestra.run.smithi138.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T04:54:46.383 INFO:teuthology.orchestra.run.smithi138.stdout:The following additional packages will be installed: 2024-09-11T04:54:46.384 INFO:teuthology.orchestra.run.smithi138.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T04:54:46.384 INFO:teuthology.orchestra.run.smithi138.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T04:54:46.384 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T04:54:46.385 INFO:teuthology.orchestra.run.smithi138.stdout:Suggested packages: 2024-09-11T04:54:46.386 INFO:teuthology.orchestra.run.smithi138.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T04:54:46.386 INFO:teuthology.orchestra.run.smithi138.stdout:Recommended packages: 2024-09-11T04:54:46.386 INFO:teuthology.orchestra.run.smithi138.stdout: thermald 2024-09-11T04:54:46.442 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2024-09-11T04:54:46.442 INFO:teuthology.orchestra.run.smithi138.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T04:54:46.443 INFO:teuthology.orchestra.run.smithi138.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T04:54:46.443 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T04:54:46.444 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages will be upgraded: 2024-09-11T04:54:46.445 INFO:teuthology.orchestra.run.smithi138.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T04:54:46.495 INFO:teuthology.orchestra.run.smithi138.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T04:54:46.495 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 113 MB of archives. 2024-09-11T04:54:46.495 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T04:54:46.495 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T04:54:46.838 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T04:54:46.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T04:54:46.910 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T04:54:47.017 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T04:54:47.096 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-09-11T04:54:47.097 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-09-11T04:54:47.149 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T04:54:47.262 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T04:54:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T04:54:47.262 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T04:54:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T04:54:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T04:54:47.263 INFO:teuthology.orchestra.run.smithi190.stdout: python2.7-minimal 2024-09-11T04:54:47.263 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T04:54:47.264 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2024-09-11T04:54:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T04:54:47.264 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T04:54:47.265 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T04:54:47.266 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2024-09-11T04:54:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T04:54:47.266 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2024-09-11T04:54:47.266 INFO:teuthology.orchestra.run.smithi190.stdout: thermald 2024-09-11T04:54:47.312 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2024-09-11T04:54:47.312 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T04:54:47.312 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T04:54:47.313 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T04:54:47.314 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages will be upgraded: 2024-09-11T04:54:47.314 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T04:54:47.493 INFO:teuthology.orchestra.run.smithi190.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T04:54:47.493 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 113 MB of archives. 2024-09-11T04:54:47.493 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T04:54:47.493 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T04:54:48.668 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T04:54:48.669 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T04:54:48.669 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T04:54:48.792 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T04:54:48.815 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T04:54:48.896 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T04:54:49.229 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 113 MB in 2s (48.0 MB/s) 2024-09-11T04:54:49.246 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T04:54:49.455 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T04:54:51.470 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T04:54:51.470 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T04:54:51.470 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T04:54:51.475 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T04:54:51.479 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:54:51.552 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:54:51.825 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T04:54:51.910 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T04:54:52.318 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 113 MB in 5s (24.7 MB/s) 2024-09-11T04:54:52.484 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T04:54:53.603 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T04:54:53.624 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:54:53.707 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:54:54.148 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T04:54:54.168 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:54:54.206 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:54:54.491 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T04:54:54.495 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:54:54.572 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:54:56.915 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T04:54:56.935 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:54:57.004 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:54:57.442 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T04:54:57.463 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:54:57.501 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:00.914 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:55:01.083 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:55:01.432 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:55:01.561 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:55:01.856 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T04:55:01.878 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T04:55:01.924 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T04:55:03.740 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:55:03.867 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:55:04.191 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:55:04.312 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:55:04.573 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T04:55:04.595 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T04:55:04.631 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T04:55:07.204 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T04:55:07.241 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:55:07.279 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:09.934 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:55:10.059 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:55:10.385 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T04:55:10.506 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:10.540 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T04:55:10.578 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T04:55:10.623 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:10.632 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T04:55:10.761 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:12.072 INFO:teuthology.orchestra.run.smithi138.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:55:12.072 INFO:teuthology.orchestra.run.smithi138.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T04:55:12.072 INFO:teuthology.orchestra.run.smithi138.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:55:12.072 INFO:teuthology.orchestra.run.smithi138.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T04:55:12.238 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:13.936 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T04:55:14.191 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:14.210 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T04:55:14.552 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T04:55:14.673 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:14.816 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T04:55:14.945 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:15.784 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T04:55:15.910 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T04:55:16.036 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:16.103 INFO:teuthology.orchestra.run.smithi138.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T04:55:16.103 INFO:teuthology.orchestra.run.smithi138.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:55:16.282 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:55:16.282 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T04:55:16.283 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:55:16.283 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T04:55:16.446 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:17.865 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:19.267 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T04:55:20.159 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T04:55:20.285 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T04:55:20.356 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T04:55:20.356 INFO:teuthology.orchestra.run.smithi190.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:55:30.519 INFO:teuthology.orchestra.run.smithi138.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T04:55:30.519 INFO:teuthology.orchestra.run.smithi138.stdout:Sourcing file `/etc/default/grub' 2024-09-11T04:55:30.539 INFO:teuthology.orchestra.run.smithi138.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T04:55:30.558 INFO:teuthology.orchestra.run.smithi138.stdout:Generating grub configuration file ... 2024-09-11T04:55:30.898 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:55:30.922 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:55:31.295 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:55:31.301 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T04:55:31.364 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T04:55:31.370 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T04:55:31.515 INFO:teuthology.orchestra.run.smithi138.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T04:55:31.515 INFO:teuthology.orchestra.run.smithi138.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T04:55:31.515 INFO:teuthology.orchestra.run.smithi138.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T04:55:31.569 INFO:teuthology.orchestra.run.smithi138.stdout:done 2024-09-11T04:55:32.087 DEBUG:teuthology.orchestra.run.smithi138:> dpkg -s linux-image-generic 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Package: linux-image-generic 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Status: install ok installed 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Priority: optional 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Section: kernel 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Installed-Size: 21 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: amd64 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Source: linux-meta 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Version: 5.15.0.119.119 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T04:55:32.147 INFO:teuthology.orchestra.run.smithi138.stdout:Recommends: thermald 2024-09-11T04:55:32.148 INFO:teuthology.orchestra.run.smithi138.stdout:Description: Generic Linux kernel image 2024-09-11T04:55:32.148 INFO:teuthology.orchestra.run.smithi138.stdout: This package will always depend on the latest generic kernel image 2024-09-11T04:55:32.148 INFO:teuthology.orchestra.run.smithi138.stdout: available. 2024-09-11T04:55:32.148 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T04:55:32.148 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T04:55:32.148 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T04:55:32.148 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T04:55:32.148 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T04:55:32.149 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get clean 2024-09-11T04:55:32.337 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2024-09-11T04:55:32.493 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T04:55:32.601 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T04:55:32.693 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T04:55:32.763 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T04:55:33.930 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-09-11T04:55:33.949 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T04:55:34.002 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-09-11T04:55:34.079 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T04:55:34.080 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub' 2024-09-11T04:55:34.090 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T04:55:34.125 INFO:teuthology.orchestra.run.smithi190.stdout:Generating grub configuration file ... 2024-09-11T04:55:34.196 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2024-09-11T04:55:34.197 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2024-09-11T04:55:34.361 INFO:teuthology.orchestra.run.smithi138.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T04:55:34.361 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T04:55:34.362 INFO:teuthology.orchestra.run.smithi138.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T04:55:34.362 INFO:teuthology.orchestra.run.smithi138.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T04:55:34.362 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T04:55:34.362 INFO:teuthology.orchestra.run.smithi138.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T04:55:34.362 INFO:teuthology.orchestra.run.smithi138.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T04:55:34.362 INFO:teuthology.orchestra.run.smithi138.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T04:55:34.396 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:55:34.403 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T04:55:34.405 DEBUG:teuthology.orchestra.run.smithi138:> dpkg -s linux-image-generic 2024-09-11T04:55:34.412 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:55:34.466 INFO:teuthology.orchestra.run.smithi138.stdout:Package: linux-image-generic 2024-09-11T04:55:34.466 INFO:teuthology.orchestra.run.smithi138.stdout:Status: install ok installed 2024-09-11T04:55:34.467 INFO:teuthology.orchestra.run.smithi138.stdout:Priority: optional 2024-09-11T04:55:34.467 INFO:teuthology.orchestra.run.smithi138.stdout:Section: kernel 2024-09-11T04:55:34.467 INFO:teuthology.orchestra.run.smithi138.stdout:Installed-Size: 21 2024-09-11T04:55:34.467 INFO:teuthology.orchestra.run.smithi138.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T04:55:34.467 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: amd64 2024-09-11T04:55:34.467 INFO:teuthology.orchestra.run.smithi138.stdout:Source: linux-meta 2024-09-11T04:55:34.468 INFO:teuthology.orchestra.run.smithi138.stdout:Version: 5.15.0.119.119 2024-09-11T04:55:34.468 INFO:teuthology.orchestra.run.smithi138.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T04:55:34.468 INFO:teuthology.orchestra.run.smithi138.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T04:55:34.468 INFO:teuthology.orchestra.run.smithi138.stdout:Recommends: thermald 2024-09-11T04:55:34.468 INFO:teuthology.orchestra.run.smithi138.stdout:Description: Generic Linux kernel image 2024-09-11T04:55:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: This package will always depend on the latest generic kernel image 2024-09-11T04:55:34.469 INFO:teuthology.orchestra.run.smithi138.stdout: available. 2024-09-11T04:55:34.469 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-09-11T04:55:34.514 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.cIh8WF2RtU 2024-09-11T04:55:34.514 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cIh8WF2RtU 2024-09-11T04:55:34.571 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.cIh8WF2RtU 2024-09-11T04:55:34.675 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:55:34.680 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T04:55:34.696 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.cIh8WF2RtU is 10KB 2024-09-11T04:55:34.710 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.cIh8WF2RtU 2024-09-11T04:55:34.715 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T04:55:34.749 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T04:55:34.754 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T04:55:34.769 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T04:55:34.770 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T04:55:34.770 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T04:55:34.837 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T04:55:34.837 DEBUG:teuthology.orchestra.run.smithi138:> sudo update-grub 2024-09-11T04:55:34.915 INFO:teuthology.orchestra.run.smithi190.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T04:55:34.915 INFO:teuthology.orchestra.run.smithi190.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T04:55:34.915 INFO:teuthology.orchestra.run.smithi190.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T04:55:34.971 INFO:teuthology.orchestra.run.smithi190.stdout:done 2024-09-11T04:55:35.406 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2024-09-11T04:55:35.461 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2024-09-11T04:55:35.461 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2024-09-11T04:55:35.461 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2024-09-11T04:55:35.461 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 21 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.119.119 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2024-09-11T04:55:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2024-09-11T04:55:35.462 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T04:55:35.463 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T04:55:35.463 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T04:55:35.463 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-11T04:55:35.463 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T04:55:35.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2024-09-11T04:55:35.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-09-11T04:55:35.924 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T04:55:35.924 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T04:55:35.993 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T04:55:36.074 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T04:55:36.076 INFO:teuthology.orchestra.run.smithi138.stderr:Sourcing file `/etc/default/grub' 2024-09-11T04:55:36.077 INFO:teuthology.orchestra.run.smithi138.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T04:55:36.081 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2024-09-11T04:55:36.340 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:55:36.347 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:55:36.660 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:55:36.666 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T04:55:36.715 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T04:55:36.721 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T04:55:36.837 INFO:teuthology.orchestra.run.smithi138.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T04:55:36.837 INFO:teuthology.orchestra.run.smithi138.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T04:55:36.837 INFO:teuthology.orchestra.run.smithi138.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T04:55:36.853 INFO:teuthology.orchestra.run.smithi138.stderr:done 2024-09-11T04:55:36.867 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2024-09-11T04:55:37.361 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T04:55:37.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T04:55:37.442 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T04:55:37.651 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-09-11T04:55:37.652 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-09-11T04:55:37.841 INFO:teuthology.orchestra.run.smithi190.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T04:55:37.841 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T04:55:37.841 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T04:55:37.841 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T04:55:37.841 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T04:55:37.842 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T04:55:37.842 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T04:55:37.842 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T04:55:37.883 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T04:55:37.885 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2024-09-11T04:55:37.896 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 21 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.119.119 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2024-09-11T04:55:37.897 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2024-09-11T04:55:37.898 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2024-09-11T04:55:37.898 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-11T04:55:37.942 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.YUISdSiUOI 2024-09-11T04:55:37.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub/grub.cfg /tmp/tmp.YUISdSiUOI 2024-09-11T04:55:37.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.YUISdSiUOI 2024-09-11T04:55:38.110 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.YUISdSiUOI is 10KB 2024-09-11T04:55:38.121 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.YUISdSiUOI 2024-09-11T04:55:38.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T04:55:38.182 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T04:55:38.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T04:55:38.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T04:55:38.249 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T04:55:38.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo update-grub 2024-09-11T04:55:39.420 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub' 2024-09-11T04:55:39.421 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T04:55:39.423 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-09-11T04:55:39.616 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T04:55:39.622 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T04:55:39.864 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T04:55:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T04:55:39.913 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T04:55:39.918 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T04:55:40.046 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T04:55:40.046 INFO:teuthology.orchestra.run.smithi190.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T04:55:40.046 INFO:teuthology.orchestra.run.smithi190.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T04:55:40.059 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-09-11T04:55:40.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-09-11T04:56:06.892 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T04:56:06.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-11T04:56:06.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:56:10.077 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-11T04:56:10.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T04:56:10.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:56:25.405 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-09-11T04:56:28.481 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-11T04:56:34.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-11T04:56:34.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:56:37.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-09-11T04:56:37.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T04:56:37.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:56:40.541 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-11T04:56:49.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-11T04:56:49.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:56:52.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T04:56:52.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T04:57:21.570 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-09-11T04:57:22.153 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-11T04:57:22.153 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-11T04:57:22.153 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-09-11T04:57:22.200 INFO:teuthology.orchestra.run.smithi138.stdout:5.15.0-119-generic 2024-09-11T04:57:22.200 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T04:57:22.201 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T04:57:22.201 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T04:57:23.201 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T04:57:23.202 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T04:57:23.273 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2024-09-11T04:57:23.302 DEBUG:teuthology.parallel:result is None 2024-09-11T04:57:24.386 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-11T04:57:24.967 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T04:57:24.967 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-119-generic"... 2024-09-11T04:57:24.968 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-11T04:57:25.015 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-119-generic 2024-09-11T04:57:25.015 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T04:57:25.015 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T04:57:25.015 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T04:57:26.016 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-11T04:57:26.016 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T04:57:26.100 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-09-11T04:57:26.128 DEBUG:teuthology.parallel:result is None 2024-09-11T04:57:26.128 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T04:57:26.138 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T04:57:26.138 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T04:57:26.141 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T04:57:26.146 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T04:57:26.204 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T04:57:26.298 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T04:57:26.298 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T04:57:26.301 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T04:57:26.392 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T04:57:26.424 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T04:57:26.425 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T04:57:26.427 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T04:57:26.451 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T04:57:26.459 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T04:57:26.472 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T04:57:26.480 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T04:57:26.482 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T04:57:26.509 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T04:57:26.509 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T04:57:26.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T04:57:26.529 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T04:57:26.640 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T04:57:26.640 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T04:57:26.642 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T04:57:26.648 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T04:57:26.692 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T04:57:26.740 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T04:57:26.740 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T04:57:26.802 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T04:57:26.808 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T04:57:26.855 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T04:57:26.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T04:57:26.927 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-09-11T04:57:26.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-09-11T04:57:27.006 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T04:57:27.015 INFO:teuthology.task.internal:Starting timer... 2024-09-11T04:57:27.015 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T04:57:27.077 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T04:57:27.198 DEBUG:teuthology.task.selinux:Excluding smithi138: OS 'ubuntu' does not support SELinux 2024-09-11T04:57:27.198 DEBUG:teuthology.task.selinux:Excluding smithi190: OS 'ubuntu' does not support SELinux 2024-09-11T04:57:27.198 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T04:57:27.198 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-11T04:57:27.198 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T04:57:27.198 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T04:57:27.290 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T04:57:27.294 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T04:57:27.403 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T04:57:27.414 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T04:57:27.414 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi138.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T05:04:56.540 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi138.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-09-11T05:04:56.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-11T05:04:56.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:04:56.626 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-09-11T05:04:56.705 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-11T05:04:56.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T05:04:56.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T05:04:56.784 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-11T05:04:56.865 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T05:04:56.865 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T05:04:56.876 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T05:04:56.877 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T05:04:56.877 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:04:56.880 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T05:04:56.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Command line: ntpd -gq 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: ---------------------------------------------------- 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: corporation. Support and training for ntp-4 are 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: available at https://www.nwtime.org/support 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: ---------------------------------------------------- 2024-09-11T05:04:56.907 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: proto: precision = 0.045 usec (-24) 2024-09-11T05:04:56.908 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: basedate set to 2022-02-04 2024-09-11T05:04:56.908 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: gps base set to 2022-02-06 (week 2196) 2024-09-11T05:04:56.908 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:04:56.908 INFO:teuthology.orchestra.run.smithi138.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:04:56.908 INFO:teuthology.orchestra.run.smithi138.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:04:56.910 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:04:56.910 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T05:04:56.910 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T05:04:56.910 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T05:04:56.910 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Listen normally on 3 enp3s0f1 172.21.15.138:123 2024-09-11T05:04:56.910 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Listen normally on 4 lo [::1]:123 2024-09-11T05:04:56.910 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f71%5]:123 2024-09-11T05:04:56.910 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:04:56 ntpd[18448]: Listening on routing socket on fd #22 for interface updates 2024-09-11T05:04:56.936 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T05:04:56.936 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Command line: ntpd -gq 2024-09-11T05:04:56.936 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: ---------------------------------------------------- 2024-09-11T05:04:56.936 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T05:04:56.936 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T05:04:56.937 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: corporation. Support and training for ntp-4 are 2024-09-11T05:04:56.937 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: available at https://www.nwtime.org/support 2024-09-11T05:04:56.937 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: ---------------------------------------------------- 2024-09-11T05:04:56.937 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: proto: precision = 0.060 usec (-24) 2024-09-11T05:04:56.937 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: basedate set to 2022-02-04 2024-09-11T05:04:56.937 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: gps base set to 2022-02-06 (week 2196) 2024-09-11T05:04:56.937 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:04:56.938 INFO:teuthology.orchestra.run.smithi190.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T05:04:56.938 INFO:teuthology.orchestra.run.smithi190.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T05:04:56.939 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Listen normally on 3 enp3s0f1 172.21.15.190:123 2024-09-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Listen normally on 4 lo [::1]:123 2024-09-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7763%5]:123 2024-09-11T05:04:56.940 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:04:56 ntpd[18453]: Listening on routing socket on fd #22 for interface updates 2024-09-11T05:04:57.907 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:57.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:57.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:04:57.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:57.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:57.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:57.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:57 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:04:57.935 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:57 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:57.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:57 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:57.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:57 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:57.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:57 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:57.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:57 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:57.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:57 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:04:58.935 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:58 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:58.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:58 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:59.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:04:59 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:04:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:04:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:04:59 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:00.907 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:00 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:00.907 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:00 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:00.935 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:00 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:00.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:00 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:00.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:00 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:00.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:00 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:00.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:00 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:00.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:00 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:01.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.910 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:01.910 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:01 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:01.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:01.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:01 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:02.935 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:02 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:02.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:02 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:02.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:02 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:02.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:02 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:02.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:02 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:02.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:02 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:03.907 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:03.907 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:03.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:03.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:03.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:03.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:03.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:03.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:03.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:03.908 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:03.909 INFO:teuthology.orchestra.run.smithi138.stdout:11 Sep 05:05:03 ntpd[18448]: ntpd: time slew -0.001729 s 2024-09-11T05:05:03.909 INFO:teuthology.orchestra.run.smithi138.stdout:ntpd: time slew -0.001729s 2024-09-11T05:05:03.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T05:05:03.909 INFO:teuthology.orchestra.run.smithi138.stderr:11 Sep 05:05:03 ntpd[18448]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T05:05:03.936 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T05:05:03.937 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 05:05:03 ntpd[18453]: ntpd: time slew -0.001078 s 2024-09-11T05:05:03.937 INFO:teuthology.orchestra.run.smithi190.stdout:ntpd: time slew -0.001078s 2024-09-11T05:05:03.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T05:05:03.937 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 05:05:03 ntpd[18453]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T05:05:03.968 INFO:teuthology.orchestra.run.smithi138.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:05:03.968 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================== 2024-09-11T05:05:03.968 INFO:teuthology.orchestra.run.smithi138.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:05:03.968 INFO:teuthology.orchestra.run.smithi138.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:05:03.968 INFO:teuthology.orchestra.run.smithi138.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:05:03.968 INFO:teuthology.orchestra.run.smithi138.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:05:03.985 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:05:03.986 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2024-09-11T05:05:03.986 INFO:teuthology.orchestra.run.smithi190.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:05:03.986 INFO:teuthology.orchestra.run.smithi190.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:05:03.986 INFO:teuthology.orchestra.run.smithi190.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:05:03.986 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T05:05:03.987 INFO:teuthology.run_tasks:Running task install... 2024-09-11T05:05:03.995 DEBUG:teuthology.task.install:project ceph 2024-09-11T05:05:03.995 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T05:05:03.995 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-11T05:05:03.995 INFO:teuthology.task.install:Using flavor: default 2024-09-11T05:05:04.004 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T05:05:04.004 INFO:teuthology.task.install:extra packages: [] 2024-09-11T05:05:04.004 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-key list | grep Ceph 2024-09-11T05:05:04.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-key list | grep Ceph 2024-09-11T05:05:04.070 INFO:teuthology.orchestra.run.smithi138.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T05:05:04.083 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T05:05:04.099 INFO:teuthology.orchestra.run.smithi138.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T05:05:04.099 INFO:teuthology.orchestra.run.smithi138.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T05:05:04.100 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-11T05:05:04.100 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T05:05:04.109 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T05:05:04.110 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T05:05:04.110 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-11T05:05:04.111 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T05:05:04.244 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default/ 2024-09-11T05:05:04.244 INFO:teuthology.task.install.deb:Package version is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T05:05:04.255 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default/ 2024-09-11T05:05:04.255 INFO:teuthology.task.install.deb:Package version is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T05:05:04.358 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:05:04.359 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T05:05:04.368 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:05:04.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T05:05:04.373 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2024-09-11T05:05:04.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-09-11T05:05:04.539 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:05:04.550 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:05:04.567 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:05:04.597 INFO:teuthology.orchestra.run.smithi138.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T05:05:04.601 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T05:05:04.622 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-11T05:05:04.622 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-11T05:05:04.644 INFO:teuthology.orchestra.run.smithi138.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T05:05:04.645 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T05:05:04.663 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:05:04.664 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-09-11T05:05:04.666 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-09-11T05:05:04.685 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T05:05:04.685 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T05:05:04.692 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-11T05:05:04.706 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-11T05:05:04.750 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T05:05:04.831 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T05:05:04.891 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 29.3 kB in 0s (86.2 kB/s) 2024-09-11T05:05:05.061 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 29.3 kB in 0s (58.9 kB/s) 2024-09-11T05:05:05.851 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-09-11T05:05:05.867 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4854-g8f219c47-1jammy cephadm=19.3.0-4854-g8f219c47-1jammy ceph-mds=19.3.0-4854-g8f219c47-1jammy ceph-mgr=19.3.0-4854-g8f219c47-1jammy ceph-common=19.3.0-4854-g8f219c47-1jammy ceph-fuse=19.3.0-4854-g8f219c47-1jammy ceph-test=19.3.0-4854-g8f219c47-1jammy ceph-volume=19.3.0-4854-g8f219c47-1jammy radosgw=19.3.0-4854-g8f219c47-1jammy python3-rados=19.3.0-4854-g8f219c47-1jammy python3-rgw=19.3.0-4854-g8f219c47-1jammy python3-cephfs=19.3.0-4854-g8f219c47-1jammy python3-rbd=19.3.0-4854-g8f219c47-1jammy libcephfs2=19.3.0-4854-g8f219c47-1jammy libcephfs-dev=19.3.0-4854-g8f219c47-1jammy librados2=19.3.0-4854-g8f219c47-1jammy librbd1=19.3.0-4854-g8f219c47-1jammy rbd-fuse=19.3.0-4854-g8f219c47-1jammy 2024-09-11T05:05:05.933 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-09-11T05:05:06.086 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T05:05:06.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4854-g8f219c47-1jammy cephadm=19.3.0-4854-g8f219c47-1jammy ceph-mds=19.3.0-4854-g8f219c47-1jammy ceph-mgr=19.3.0-4854-g8f219c47-1jammy ceph-common=19.3.0-4854-g8f219c47-1jammy ceph-fuse=19.3.0-4854-g8f219c47-1jammy ceph-test=19.3.0-4854-g8f219c47-1jammy ceph-volume=19.3.0-4854-g8f219c47-1jammy radosgw=19.3.0-4854-g8f219c47-1jammy python3-rados=19.3.0-4854-g8f219c47-1jammy python3-rgw=19.3.0-4854-g8f219c47-1jammy python3-cephfs=19.3.0-4854-g8f219c47-1jammy python3-rbd=19.3.0-4854-g8f219c47-1jammy libcephfs2=19.3.0-4854-g8f219c47-1jammy libcephfs-dev=19.3.0-4854-g8f219c47-1jammy librados2=19.3.0-4854-g8f219c47-1jammy librbd1=19.3.0-4854-g8f219c47-1jammy rbd-fuse=19.3.0-4854-g8f219c47-1jammy 2024-09-11T05:05:06.152 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2024-09-11T05:05:06.153 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2024-09-11T05:05:06.165 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T05:05:06.365 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:05:06.365 INFO:teuthology.orchestra.run.smithi138.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:05:06.365 INFO:teuthology.orchestra.run.smithi138.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:05:06.366 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:05:06.366 INFO:teuthology.orchestra.run.smithi138.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:05:06.366 INFO:teuthology.orchestra.run.smithi138.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:05:06.366 INFO:teuthology.orchestra.run.smithi138.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:05:06.367 INFO:teuthology.orchestra.run.smithi138.stdout:The following additional packages will be installed: 2024-09-11T05:05:06.367 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T05:05:06.367 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T05:05:06.367 INFO:teuthology.orchestra.run.smithi138.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-11T05:05:06.367 INFO:teuthology.orchestra.run.smithi138.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-11T05:05:06.368 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-11T05:05:06.368 INFO:teuthology.orchestra.run.smithi138.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-11T05:05:06.368 INFO:teuthology.orchestra.run.smithi138.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-11T05:05:06.368 INFO:teuthology.orchestra.run.smithi138.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T05:05:06.369 INFO:teuthology.orchestra.run.smithi138.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T05:05:06.370 INFO:teuthology.orchestra.run.smithi138.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T05:05:06.370 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T05:05:06.370 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet zip 2024-09-11T05:05:06.370 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-09-11T05:05:06.370 INFO:teuthology.orchestra.run.smithi138.stdout:Suggested packages: 2024-09-11T05:05:06.370 INFO:teuthology.orchestra.run.smithi138.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T05:05:06.370 INFO:teuthology.orchestra.run.smithi138.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T05:05:06.370 INFO:teuthology.orchestra.run.smithi138.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T05:05:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T05:05:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T05:05:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T05:05:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T05:05:06.371 INFO:teuthology.orchestra.run.smithi138.stdout:Recommended packages: 2024-09-11T05:05:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: btrfs-tools 2024-09-11T05:05:06.371 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-09-11T05:05:06.473 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2024-09-11T05:05:06.473 INFO:teuthology.orchestra.run.smithi138.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T05:05:06.473 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T05:05:06.474 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:05:06.475 INFO:teuthology.orchestra.run.smithi138.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T05:05:06.476 INFO:teuthology.orchestra.run.smithi138.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-11T05:05:06.535 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T05:05:06.535 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 251 MB of archives. 2024-09-11T05:05:06.536 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 934 MB of additional disk space will be used. 2024-09-11T05:05:06.536 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T05:05:06.580 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T05:05:06.580 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T05:05:06.580 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T05:05:06.580 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T05:05:06.581 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T05:05:06.581 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T05:05:06.581 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T05:05:06.581 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2024-09-11T05:05:06.581 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T05:05:06.582 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T05:05:06.582 INFO:teuthology.orchestra.run.smithi190.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-11T05:05:06.582 INFO:teuthology.orchestra.run.smithi190.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-11T05:05:06.582 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-11T05:05:06.583 INFO:teuthology.orchestra.run.smithi190.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-11T05:05:06.583 INFO:teuthology.orchestra.run.smithi190.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-11T05:05:06.583 INFO:teuthology.orchestra.run.smithi190.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-11T05:05:06.583 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-11T05:05:06.583 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-11T05:05:06.583 INFO:teuthology.orchestra.run.smithi190.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-11T05:05:06.583 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T05:05:06.584 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet zip 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2024-09-11T05:05:06.585 INFO:teuthology.orchestra.run.smithi190.stdout: btrfs-tools 2024-09-11T05:05:06.618 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4854-g8f219c47-1jammy [3,633 kB] 2024-09-11T05:05:06.624 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T05:05:06.627 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T05:05:06.644 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T05:05:06.689 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2024-09-11T05:05:06.689 INFO:teuthology.orchestra.run.smithi190.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T05:05:06.689 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T05:05:06.689 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T05:05:06.689 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T05:05:06.689 INFO:teuthology.orchestra.run.smithi190.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-11T05:05:06.689 INFO:teuthology.orchestra.run.smithi190.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T05:05:06.690 INFO:teuthology.orchestra.run.smithi190.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T05:05:06.691 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T05:05:06.692 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T05:05:06.692 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T05:05:06.692 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T05:05:06.692 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T05:05:06.692 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T05:05:06.692 INFO:teuthology.orchestra.run.smithi190.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-11T05:05:06.702 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T05:05:06.707 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T05:05:06.719 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T05:05:06.723 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-11T05:05:06.725 INFO:teuthology.orchestra.run.smithi138.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T05:05:06.725 INFO:teuthology.orchestra.run.smithi138.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T05:05:06.726 INFO:teuthology.orchestra.run.smithi138.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T05:05:06.734 INFO:teuthology.orchestra.run.smithi138.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T05:05:06.737 INFO:teuthology.orchestra.run.smithi138.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T05:05:06.739 INFO:teuthology.orchestra.run.smithi138.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T05:05:06.740 INFO:teuthology.orchestra.run.smithi138.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T05:05:06.741 INFO:teuthology.orchestra.run.smithi138.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T05:05:06.748 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T05:05:06.749 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 251 MB of archives. 2024-09-11T05:05:06.749 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 934 MB of additional disk space will be used. 2024-09-11T05:05:06.749 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T05:05:06.758 INFO:teuthology.orchestra.run.smithi138.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T05:05:06.760 INFO:teuthology.orchestra.run.smithi138.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T05:05:06.762 INFO:teuthology.orchestra.run.smithi138.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T05:05:06.763 INFO:teuthology.orchestra.run.smithi138.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T05:05:06.763 INFO:teuthology.orchestra.run.smithi138.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T05:05:06.764 INFO:teuthology.orchestra.run.smithi138.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T05:05:06.764 INFO:teuthology.orchestra.run.smithi138.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T05:05:06.765 INFO:teuthology.orchestra.run.smithi138.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T05:05:06.775 INFO:teuthology.orchestra.run.smithi138.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T05:05:06.776 INFO:teuthology.orchestra.run.smithi138.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T05:05:06.796 INFO:teuthology.orchestra.run.smithi138.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T05:05:06.797 INFO:teuthology.orchestra.run.smithi138.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T05:05:06.799 INFO:teuthology.orchestra.run.smithi138.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T05:05:06.800 INFO:teuthology.orchestra.run.smithi138.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T05:05:06.800 INFO:teuthology.orchestra.run.smithi138.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T05:05:06.801 INFO:teuthology.orchestra.run.smithi138.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T05:05:06.802 INFO:teuthology.orchestra.run.smithi138.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T05:05:06.802 INFO:teuthology.orchestra.run.smithi138.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T05:05:06.805 INFO:teuthology.orchestra.run.smithi138.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4854-g8f219c47-1jammy [3,330 kB] 2024-09-11T05:05:06.815 INFO:teuthology.orchestra.run.smithi138.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T05:05:06.816 INFO:teuthology.orchestra.run.smithi138.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T05:05:06.832 INFO:teuthology.orchestra.run.smithi138.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T05:05:06.832 INFO:teuthology.orchestra.run.smithi138.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T05:05:06.838 INFO:teuthology.orchestra.run.smithi138.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T05:05:06.838 INFO:teuthology.orchestra.run.smithi138.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T05:05:06.838 INFO:teuthology.orchestra.run.smithi138.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T05:05:06.839 INFO:teuthology.orchestra.run.smithi138.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T05:05:06.840 INFO:teuthology.orchestra.run.smithi138.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T05:05:06.841 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T05:05:06.842 INFO:teuthology.orchestra.run.smithi138.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T05:05:06.844 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T05:05:06.849 INFO:teuthology.orchestra.run.smithi138.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T05:05:06.850 INFO:teuthology.orchestra.run.smithi138.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T05:05:06.853 INFO:teuthology.orchestra.run.smithi138.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4854-g8f219c47-1jammy [721 kB] 2024-09-11T05:05:06.860 INFO:teuthology.orchestra.run.smithi138.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T05:05:06.865 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T05:05:06.865 INFO:teuthology.orchestra.run.smithi138.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T05:05:06.865 INFO:teuthology.orchestra.run.smithi138.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4854-g8f219c47-1jammy [356 kB] 2024-09-11T05:05:06.865 INFO:teuthology.orchestra.run.smithi138.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T05:05:06.865 INFO:teuthology.orchestra.run.smithi138.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T05:05:06.866 INFO:teuthology.orchestra.run.smithi138.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T05:05:06.869 INFO:teuthology.orchestra.run.smithi138.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4854-g8f219c47-1jammy [32.8 kB] 2024-09-11T05:05:06.870 INFO:teuthology.orchestra.run.smithi138.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4854-g8f219c47-1jammy [180 kB] 2024-09-11T05:05:06.873 INFO:teuthology.orchestra.run.smithi138.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4854-g8f219c47-1jammy [71.2 kB] 2024-09-11T05:05:06.873 INFO:teuthology.orchestra.run.smithi138.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4854-g8f219c47-1jammy [338 kB] 2024-09-11T05:05:06.878 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4854-g8f219c47-1jammy [3,633 kB] 2024-09-11T05:05:06.880 INFO:teuthology.orchestra.run.smithi138.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4854-g8f219c47-1jammy [7,988 kB] 2024-09-11T05:05:06.918 INFO:teuthology.orchestra.run.smithi138.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T05:05:06.919 INFO:teuthology.orchestra.run.smithi138.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T05:05:06.920 INFO:teuthology.orchestra.run.smithi138.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T05:05:06.920 INFO:teuthology.orchestra.run.smithi138.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T05:05:06.921 INFO:teuthology.orchestra.run.smithi138.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T05:05:06.922 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T05:05:06.927 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T05:05:06.938 INFO:teuthology.orchestra.run.smithi138.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T05:05:06.939 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T05:05:06.940 INFO:teuthology.orchestra.run.smithi138.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T05:05:06.940 INFO:teuthology.orchestra.run.smithi138.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T05:05:06.943 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-11T05:05:06.945 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T05:05:06.945 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T05:05:06.946 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T05:05:06.954 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T05:05:06.955 INFO:teuthology.orchestra.run.smithi138.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T05:05:06.955 INFO:teuthology.orchestra.run.smithi138.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T05:05:06.955 INFO:teuthology.orchestra.run.smithi138.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T05:05:06.956 INFO:teuthology.orchestra.run.smithi138.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T05:05:06.956 INFO:teuthology.orchestra.run.smithi138.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T05:05:06.957 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T05:05:06.957 INFO:teuthology.orchestra.run.smithi138.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T05:05:06.958 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T05:05:06.961 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T05:05:06.961 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T05:05:06.972 INFO:teuthology.orchestra.run.smithi138.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T05:05:06.972 INFO:teuthology.orchestra.run.smithi138.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T05:05:06.975 INFO:teuthology.orchestra.run.smithi138.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T05:05:06.978 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T05:05:06.980 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T05:05:06.982 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T05:05:06.982 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T05:05:06.983 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T05:05:06.984 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T05:05:06.984 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T05:05:06.984 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T05:05:06.988 INFO:teuthology.orchestra.run.smithi138.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T05:05:06.990 INFO:teuthology.orchestra.run.smithi138.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T05:05:06.990 INFO:teuthology.orchestra.run.smithi138.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T05:05:06.993 INFO:teuthology.orchestra.run.smithi138.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T05:05:06.995 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T05:05:06.996 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T05:05:06.997 INFO:teuthology.orchestra.run.smithi138.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T05:05:06.997 INFO:teuthology.orchestra.run.smithi138.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T05:05:06.997 INFO:teuthology.orchestra.run.smithi138.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T05:05:07.006 INFO:teuthology.orchestra.run.smithi138.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T05:05:07.007 INFO:teuthology.orchestra.run.smithi138.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T05:05:07.010 INFO:teuthology.orchestra.run.smithi138.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T05:05:07.012 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T05:05:07.012 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T05:05:07.014 INFO:teuthology.orchestra.run.smithi138.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4854-g8f219c47-1jammy [112 kB] 2024-09-11T05:05:07.015 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T05:05:07.015 INFO:teuthology.orchestra.run.smithi138.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4854-g8f219c47-1jammy [473 kB] 2024-09-11T05:05:07.015 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T05:05:07.016 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T05:05:07.016 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T05:05:07.017 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T05:05:07.017 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T05:05:07.023 INFO:teuthology.orchestra.run.smithi138.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T05:05:07.023 INFO:teuthology.orchestra.run.smithi138.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4854-g8f219c47-1jammy [28.3 MB] 2024-09-11T05:05:07.024 INFO:teuthology.orchestra.run.smithi138.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T05:05:07.024 INFO:teuthology.orchestra.run.smithi138.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T05:05:07.026 INFO:teuthology.orchestra.run.smithi138.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T05:05:07.026 INFO:teuthology.orchestra.run.smithi138.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T05:05:07.029 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T05:05:07.030 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T05:05:07.032 INFO:teuthology.orchestra.run.smithi138.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T05:05:07.032 INFO:teuthology.orchestra.run.smithi138.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T05:05:07.040 INFO:teuthology.orchestra.run.smithi138.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T05:05:07.042 INFO:teuthology.orchestra.run.smithi138.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T05:05:07.043 INFO:teuthology.orchestra.run.smithi138.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T05:05:07.045 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T05:05:07.046 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T05:05:07.051 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T05:05:07.051 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T05:05:07.052 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T05:05:07.053 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T05:05:07.054 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T05:05:07.055 INFO:teuthology.orchestra.run.smithi190.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T05:05:07.063 INFO:teuthology.orchestra.run.smithi190.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T05:05:07.064 INFO:teuthology.orchestra.run.smithi190.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T05:05:07.074 INFO:teuthology.orchestra.run.smithi190.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T05:05:07.076 INFO:teuthology.orchestra.run.smithi190.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T05:05:07.077 INFO:teuthology.orchestra.run.smithi190.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T05:05:07.077 INFO:teuthology.orchestra.run.smithi190.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T05:05:07.080 INFO:teuthology.orchestra.run.smithi190.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T05:05:07.133 INFO:teuthology.orchestra.run.smithi190.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T05:05:07.134 INFO:teuthology.orchestra.run.smithi190.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T05:05:07.134 INFO:teuthology.orchestra.run.smithi190.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T05:05:07.134 INFO:teuthology.orchestra.run.smithi190.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T05:05:07.135 INFO:teuthology.orchestra.run.smithi190.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T05:05:07.153 INFO:teuthology.orchestra.run.smithi190.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T05:05:07.155 INFO:teuthology.orchestra.run.smithi190.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T05:05:07.155 INFO:teuthology.orchestra.run.smithi190.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T05:05:07.169 INFO:teuthology.orchestra.run.smithi190.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T05:05:07.170 INFO:teuthology.orchestra.run.smithi190.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T05:05:07.170 INFO:teuthology.orchestra.run.smithi190.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T05:05:07.170 INFO:teuthology.orchestra.run.smithi190.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T05:05:07.171 INFO:teuthology.orchestra.run.smithi190.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T05:05:07.171 INFO:teuthology.orchestra.run.smithi190.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T05:05:07.185 INFO:teuthology.orchestra.run.smithi190.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T05:05:07.185 INFO:teuthology.orchestra.run.smithi190.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T05:05:07.188 INFO:teuthology.orchestra.run.smithi190.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T05:05:07.202 INFO:teuthology.orchestra.run.smithi190.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T05:05:07.203 INFO:teuthology.orchestra.run.smithi190.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T05:05:07.204 INFO:teuthology.orchestra.run.smithi190.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T05:05:07.207 INFO:teuthology.orchestra.run.smithi190.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T05:05:07.209 INFO:teuthology.orchestra.run.smithi190.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T05:05:07.210 INFO:teuthology.orchestra.run.smithi190.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T05:05:07.210 INFO:teuthology.orchestra.run.smithi190.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T05:05:07.219 INFO:teuthology.orchestra.run.smithi190.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T05:05:07.219 INFO:teuthology.orchestra.run.smithi190.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T05:05:07.222 INFO:teuthology.orchestra.run.smithi190.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T05:05:07.236 INFO:teuthology.orchestra.run.smithi190.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T05:05:07.237 INFO:teuthology.orchestra.run.smithi190.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T05:05:07.237 INFO:teuthology.orchestra.run.smithi190.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T05:05:07.238 INFO:teuthology.orchestra.run.smithi190.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T05:05:07.239 INFO:teuthology.orchestra.run.smithi190.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T05:05:07.245 INFO:teuthology.orchestra.run.smithi190.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T05:05:07.245 INFO:teuthology.orchestra.run.smithi190.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T05:05:07.253 INFO:teuthology.orchestra.run.smithi190.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T05:05:07.255 INFO:teuthology.orchestra.run.smithi190.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T05:05:07.255 INFO:teuthology.orchestra.run.smithi190.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T05:05:07.680 INFO:teuthology.orchestra.run.smithi138.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4854-g8f219c47-1jammy [5,206 kB] 2024-09-11T05:05:07.746 INFO:teuthology.orchestra.run.smithi138.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4854-g8f219c47-1jammy [251 kB] 2024-09-11T05:05:07.749 INFO:teuthology.orchestra.run.smithi138.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4854-g8f219c47-1jammy [124 kB] 2024-09-11T05:05:07.750 INFO:teuthology.orchestra.run.smithi138.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4854-g8f219c47-1jammy [1,562 kB] 2024-09-11T05:05:07.764 INFO:teuthology.orchestra.run.smithi138.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4854-g8f219c47-1jammy [6,308 kB] 2024-09-11T05:05:07.781 INFO:teuthology.orchestra.run.smithi190.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4854-g8f219c47-1jammy [3,330 kB] 2024-09-11T05:05:07.869 INFO:teuthology.orchestra.run.smithi138.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4854-g8f219c47-1jammy [23.1 MB] 2024-09-11T05:05:08.294 INFO:teuthology.orchestra.run.smithi190.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4854-g8f219c47-1jammy [721 kB] 2024-09-11T05:05:08.367 INFO:teuthology.orchestra.run.smithi190.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4854-g8f219c47-1jammy [356 kB] 2024-09-11T05:05:08.375 INFO:teuthology.orchestra.run.smithi190.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4854-g8f219c47-1jammy [32.8 kB] 2024-09-11T05:05:08.384 INFO:teuthology.orchestra.run.smithi190.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4854-g8f219c47-1jammy [180 kB] 2024-09-11T05:05:08.390 INFO:teuthology.orchestra.run.smithi190.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4854-g8f219c47-1jammy [71.2 kB] 2024-09-11T05:05:08.392 INFO:teuthology.orchestra.run.smithi190.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4854-g8f219c47-1jammy [338 kB] 2024-09-11T05:05:08.398 INFO:teuthology.orchestra.run.smithi138.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4854-g8f219c47-1jammy [14.0 kB] 2024-09-11T05:05:08.398 INFO:teuthology.orchestra.run.smithi138.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4854-g8f219c47-1jammy [880 kB] 2024-09-11T05:05:08.408 INFO:teuthology.orchestra.run.smithi190.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4854-g8f219c47-1jammy [7,988 kB] 2024-09-11T05:05:08.411 INFO:teuthology.orchestra.run.smithi138.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4854-g8f219c47-1jammy [2,454 kB] 2024-09-11T05:05:08.447 INFO:teuthology.orchestra.run.smithi138.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4854-g8f219c47-1jammy [813 kB] 2024-09-11T05:05:08.457 INFO:teuthology.orchestra.run.smithi138.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4854-g8f219c47-1jammy [160 kB] 2024-09-11T05:05:08.459 INFO:teuthology.orchestra.run.smithi138.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4854-g8f219c47-1jammy [61.4 MB] 2024-09-11T05:05:08.998 INFO:teuthology.orchestra.run.smithi190.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4854-g8f219c47-1jammy [112 kB] 2024-09-11T05:05:09.013 INFO:teuthology.orchestra.run.smithi190.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4854-g8f219c47-1jammy [473 kB] 2024-09-11T05:05:09.051 INFO:teuthology.orchestra.run.smithi190.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4854-g8f219c47-1jammy [28.3 MB] 2024-09-11T05:05:10.150 INFO:teuthology.orchestra.run.smithi138.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4854-g8f219c47-1jammy [8,625 kB] 2024-09-11T05:05:10.193 INFO:teuthology.orchestra.run.smithi190.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4854-g8f219c47-1jammy [5,206 kB] 2024-09-11T05:05:10.269 INFO:teuthology.orchestra.run.smithi190.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4854-g8f219c47-1jammy [251 kB] 2024-09-11T05:05:10.271 INFO:teuthology.orchestra.run.smithi190.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4854-g8f219c47-1jammy [124 kB] 2024-09-11T05:05:10.272 INFO:teuthology.orchestra.run.smithi190.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4854-g8f219c47-1jammy [1,562 kB] 2024-09-11T05:05:10.282 INFO:teuthology.orchestra.run.smithi138.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4854-g8f219c47-1jammy [14.2 kB] 2024-09-11T05:05:10.283 INFO:teuthology.orchestra.run.smithi138.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4854-g8f219c47-1jammy [55.8 MB] 2024-09-11T05:05:10.420 INFO:teuthology.orchestra.run.smithi190.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4854-g8f219c47-1jammy [6,308 kB] 2024-09-11T05:05:10.539 INFO:teuthology.orchestra.run.smithi190.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4854-g8f219c47-1jammy [23.1 MB] 2024-09-11T05:05:11.287 INFO:teuthology.orchestra.run.smithi190.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4854-g8f219c47-1jammy [14.0 kB] 2024-09-11T05:05:11.288 INFO:teuthology.orchestra.run.smithi190.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4854-g8f219c47-1jammy [880 kB] 2024-09-11T05:05:11.305 INFO:teuthology.orchestra.run.smithi190.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4854-g8f219c47-1jammy [2,454 kB] 2024-09-11T05:05:11.346 INFO:teuthology.orchestra.run.smithi190.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4854-g8f219c47-1jammy [813 kB] 2024-09-11T05:05:11.355 INFO:teuthology.orchestra.run.smithi190.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4854-g8f219c47-1jammy [160 kB] 2024-09-11T05:05:11.357 INFO:teuthology.orchestra.run.smithi190.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4854-g8f219c47-1jammy [61.4 MB] 2024-09-11T05:05:12.525 INFO:teuthology.orchestra.run.smithi138.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4854-g8f219c47-1jammy [130 kB] 2024-09-11T05:05:12.528 INFO:teuthology.orchestra.run.smithi138.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4854-g8f219c47-1jammy [39.0 kB] 2024-09-11T05:05:12.529 INFO:teuthology.orchestra.run.smithi138.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4854-g8f219c47-1jammy [15.9 MB] 2024-09-11T05:05:13.062 INFO:teuthology.orchestra.run.smithi138.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4854-g8f219c47-1jammy [91.2 kB] 2024-09-11T05:05:13.376 INFO:teuthology.orchestra.run.smithi190.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4854-g8f219c47-1jammy [8,625 kB] 2024-09-11T05:05:13.484 INFO:teuthology.orchestra.run.smithi190.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4854-g8f219c47-1jammy [14.2 kB] 2024-09-11T05:05:13.485 INFO:teuthology.orchestra.run.smithi190.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4854-g8f219c47-1jammy [55.8 MB] 2024-09-11T05:05:13.633 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 251 MB in 7s (38.2 MB/s) 2024-09-11T05:05:13.801 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T05:05:13.901 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T05:05:13.906 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T05:05:13.991 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:05:14.336 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T05:05:14.362 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T05:05:14.402 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:05:14.639 INFO:teuthology.orchestra.run.smithi190.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4854-g8f219c47-1jammy [130 kB] 2024-09-11T05:05:14.645 INFO:teuthology.orchestra.run.smithi190.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4854-g8f219c47-1jammy [39.0 kB] 2024-09-11T05:05:14.707 INFO:teuthology.orchestra.run.smithi190.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4854-g8f219c47-1jammy [15.9 MB] 2024-09-11T05:05:14.730 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T05:05:14.757 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T05:05:14.797 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:05:15.021 INFO:teuthology.orchestra.run.smithi190.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4854-g8f219c47-1jammy [91.2 kB] 2024-09-11T05:05:15.251 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T05:05:15.277 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:05:15.326 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:05:15.598 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 251 MB in 8s (30.2 MB/s) 2024-09-11T05:05:15.762 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T05:05:15.778 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T05:05:15.805 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:05:15.845 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:05:15.862 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T05:05:15.868 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T05:05:15.952 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:05:16.190 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T05:05:16.216 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:05:16.257 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:05:16.289 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T05:05:16.315 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T05:05:16.355 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:05:16.601 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T05:05:16.628 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T05:05:16.668 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:05:16.692 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T05:05:16.719 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T05:05:16.758 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:05:17.013 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package librados2. 2024-09-11T05:05:17.039 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../007-librados2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:17.079 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:17.220 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T05:05:17.247 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:05:17.303 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:05:17.516 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libnbd0. 2024-09-11T05:05:17.531 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-11T05:05:17.565 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-11T05:05:17.773 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T05:05:17.799 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:05:17.840 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:05:17.885 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package librbd1. 2024-09-11T05:05:17.911 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../009-librbd1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:18.030 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:18.185 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T05:05:18.211 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T05:05:18.251 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:05:18.396 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T05:05:18.422 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:18.463 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:18.621 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T05:05:18.647 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T05:05:18.695 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:05:18.740 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-rados. 2024-09-11T05:05:18.756 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:18.790 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:19.082 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librados2. 2024-09-11T05:05:19.084 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T05:05:19.108 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../007-librados2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:19.110 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:19.151 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:19.156 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:19.403 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T05:05:19.418 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:19.461 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:19.781 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T05:05:19.807 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:19.843 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libnbd0. 2024-09-11T05:05:19.847 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:19.869 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-11T05:05:19.909 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-11T05:05:20.167 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T05:05:20.193 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T05:05:20.220 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librbd1. 2024-09-11T05:05:20.234 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:05:20.246 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../009-librbd1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:20.287 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:20.520 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T05:05:20.546 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T05:05:20.595 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T05:05:20.765 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T05:05:20.791 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:20.831 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T05:05:20.839 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:20.846 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:20.889 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:21.150 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rados. 2024-09-11T05:05:21.165 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:21.208 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:21.234 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T05:05:21.260 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T05:05:21.300 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:05:21.528 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T05:05:21.554 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:21.587 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T05:05:21.594 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:21.601 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T05:05:21.644 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:05:21.847 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T05:05:21.862 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:21.905 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:21.939 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T05:05:21.954 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T05:05:21.989 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:05:22.291 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T05:05:22.300 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package lua5.1. 2024-09-11T05:05:22.318 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:22.327 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T05:05:22.366 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:22.367 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:05:22.695 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package lua-any. 2024-09-11T05:05:22.710 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-11T05:05:22.736 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T05:05:22.753 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T05:05:22.762 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T05:05:22.811 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:05:22.967 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package zip. 2024-09-11T05:05:22.993 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-11T05:05:23.033 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T05:05:23.130 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T05:05:23.157 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T05:05:23.197 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T05:05:23.354 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package unzip. 2024-09-11T05:05:23.380 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T05:05:23.420 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:05:23.457 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T05:05:23.473 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:23.516 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:23.817 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package luarocks. 2024-09-11T05:05:23.843 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T05:05:23.869 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T05:05:23.883 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:05:23.895 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T05:05:23.943 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:05:24.226 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package librgw2. 2024-09-11T05:05:24.241 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../026-librgw2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:24.263 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T05:05:24.278 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T05:05:24.284 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:24.329 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:05:24.657 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T05:05:24.673 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T05:05:24.721 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:05:24.929 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T05:05:24.944 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:24.987 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:25.161 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua5.1. 2024-09-11T05:05:25.187 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T05:05:25.227 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:05:25.580 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-any. 2024-09-11T05:05:25.606 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-11T05:05:25.624 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T05:05:25.646 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T05:05:25.650 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T05:05:25.690 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:05:25.899 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package zip. 2024-09-11T05:05:25.914 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-11T05:05:25.957 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T05:05:25.969 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T05:05:25.995 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:26.036 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:26.302 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package unzip. 2024-09-11T05:05:26.329 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T05:05:26.369 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:05:26.464 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-common. 2024-09-11T05:05:26.491 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:26.531 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:26.739 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package luarocks. 2024-09-11T05:05:26.765 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T05:05:26.797 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:05:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librgw2. 2024-09-11T05:05:27.173 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../026-librgw2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T05:05:27.842 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:27.890 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:28.029 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-base. 2024-09-11T05:05:28.055 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:28.097 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:28.235 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T05:05:28.262 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T05:05:28.301 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:05:28.571 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T05:05:28.597 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:28.629 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:28.692 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T05:05:28.719 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T05:05:28.759 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:05:29.040 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-common. 2024-09-11T05:05:29.054 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T05:05:29.067 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:29.080 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T05:05:29.109 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:29.120 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:05:29.457 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T05:05:29.483 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T05:05:29.524 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:05:29.786 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T05:05:29.812 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T05:05:29.844 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:05:30.147 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T05:05:30.173 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T05:05:30.214 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:05:30.526 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T05:05:30.552 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T05:05:30.592 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T05:05:30.747 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-base. 2024-09-11T05:05:30.773 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:30.831 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:30.862 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-portend. 2024-09-11T05:05:30.888 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-11T05:05:30.920 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T05:05:31.232 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T05:05:31.259 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T05:05:31.299 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T05:05:31.426 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T05:05:31.453 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T05:05:31.493 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:05:31.602 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T05:05:31.629 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T05:05:31.669 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:05:31.796 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T05:05:31.822 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T05:05:31.862 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:05:32.015 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T05:05:32.041 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T05:05:32.106 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T05:05:32.182 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T05:05:32.209 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T05:05:32.248 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:05:32.418 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T05:05:32.445 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T05:05:32.485 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T05:05:32.576 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T05:05:32.603 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T05:05:32.651 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:05:32.763 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T05:05:32.778 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T05:05:32.813 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:05:32.970 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T05:05:32.997 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T05:05:33.037 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:05:33.092 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-mako. 2024-09-11T05:05:33.118 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T05:05:33.158 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:05:33.340 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T05:05:33.366 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T05:05:33.406 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T05:05:33.487 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T05:05:33.513 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T05:05:33.553 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:05:33.743 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-portend. 2024-09-11T05:05:33.769 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-11T05:05:33.809 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T05:05:33.823 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T05:05:33.850 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T05:05:33.882 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:05:34.179 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T05:05:34.185 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-webob. 2024-09-11T05:05:34.205 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T05:05:34.212 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T05:05:34.252 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:05:34.253 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T05:05:34.581 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T05:05:34.605 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T05:05:34.608 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T05:05:34.632 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T05:05:34.648 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:05:34.674 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:05:34.984 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T05:05:34.985 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T05:05:35.010 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T05:05:35.011 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T05:05:35.043 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T05:05:35.050 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:05:35.312 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-paste. 2024-09-11T05:05:35.327 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T05:05:35.362 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:05:35.396 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T05:05:35.422 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T05:05:35.463 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T05:05:35.765 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T05:05:35.791 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T05:05:35.831 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:05:35.866 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T05:05:35.892 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T05:05:35.958 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:05:36.101 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-mako. 2024-09-11T05:05:36.128 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T05:05:36.159 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:05:36.236 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T05:05:36.263 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T05:05:36.303 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:05:36.471 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T05:05:36.497 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T05:05:36.537 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:05:36.581 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T05:05:36.608 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T05:05:36.647 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T05:05:36.840 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T05:05:36.867 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T05:05:36.907 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:05:36.959 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T05:05:36.986 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T05:05:37.026 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:05:37.193 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webob. 2024-09-11T05:05:37.219 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T05:05:37.259 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:05:37.398 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T05:05:37.425 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T05:05:37.465 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T05:05:37.637 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T05:05:37.664 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T05:05:37.707 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:05:37.785 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-idna. 2024-09-11T05:05:37.811 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T05:05:37.852 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:05:37.973 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T05:05:37.988 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T05:05:38.023 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:05:38.155 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T05:05:38.181 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T05:05:38.222 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:05:38.309 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-paste. 2024-09-11T05:05:38.336 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T05:05:38.384 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:05:38.525 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-requests. 2024-09-11T05:05:38.552 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T05:05:38.592 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:05:38.704 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T05:05:38.730 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T05:05:38.770 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:05:38.904 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T05:05:38.930 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T05:05:38.970 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:05:39.048 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T05:05:39.075 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T05:05:39.115 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:05:39.249 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T05:05:39.261 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:39.299 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:39.418 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T05:05:39.444 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T05:05:39.493 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T05:05:39.671 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T05:05:39.698 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:39.738 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:39.787 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T05:05:39.813 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T05:05:40.104 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:05:40.909 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T05:05:40.935 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T05:05:40.958 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T05:05:40.975 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:05:40.985 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T05:05:41.025 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T05:05:41.304 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T05:05:41.328 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-idna. 2024-09-11T05:05:41.330 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T05:05:41.354 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T05:05:41.370 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T05:05:41.411 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:05:41.674 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T05:05:41.701 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:41.705 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T05:05:41.732 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T05:05:41.732 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:41.772 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:05:42.075 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests. 2024-09-11T05:05:42.101 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T05:05:42.141 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:05:42.152 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T05:05:42.179 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:42.219 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:42.461 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T05:05:42.487 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T05:05:42.553 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:05:42.814 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T05:05:42.829 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:42.872 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:42.873 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T05:05:42.900 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:42.940 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:43.252 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T05:05:43.279 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:43.318 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:43.579 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T05:05:43.606 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T05:05:43.638 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:05:43.932 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T05:05:43.959 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T05:05:43.998 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T05:05:44.027 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph. 2024-09-11T05:05:44.043 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../067-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:44.085 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:44.352 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T05:05:44.378 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:44.405 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T05:05:44.418 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:44.431 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:44.471 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:44.813 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T05:05:44.825 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T05:05:44.839 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:44.840 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:44.879 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:44.883 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:45.287 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package cephadm. 2024-09-11T05:05:45.313 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../070-cephadm_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:45.353 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:45.525 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T05:05:45.551 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:45.591 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:45.673 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T05:05:45.700 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T05:05:45.739 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T05:05:46.068 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T05:05:46.083 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:05:46.126 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:05:46.628 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph. 2024-09-11T05:05:46.643 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../067-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:46.686 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:47.005 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T05:05:47.032 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:47.071 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:47.299 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-babel. 2024-09-11T05:05:47.326 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:05:47.357 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:05:47.408 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T05:05:47.434 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:47.465 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:47.694 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T05:05:47.721 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T05:05:47.761 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:05:47.835 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package cephadm. 2024-09-11T05:05:47.862 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../070-cephadm_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:05:47.902 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:48.081 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T05:05:48.107 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:48.147 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:48.221 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T05:05:48.248 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T05:05:48.288 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T05:05:48.451 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T05:05:48.478 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T05:05:48.518 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T05:05:48.607 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T05:05:48.622 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:05:48.665 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:05:48.821 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-routes. 2024-09-11T05:05:48.848 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T05:05:48.888 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:05:49.208 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T05:05:49.235 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:49.274 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:49.654 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-babel. 2024-09-11T05:05:49.681 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T05:05:49.720 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:05:50.023 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T05:05:50.043 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T05:05:50.081 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:05:50.409 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T05:05:50.436 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:50.475 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:50.804 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T05:05:50.831 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T05:05:50.870 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T05:05:51.131 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-routes. 2024-09-11T05:05:51.147 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T05:05:51.181 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:05:51.526 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T05:05:51.553 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:51.592 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:51.974 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T05:05:52.003 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T05:05:52.040 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:05:52.556 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T05:05:52.580 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T05:05:52.619 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:05:52.973 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T05:05:53.001 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T05:05:53.039 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:05:53.351 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T05:05:53.378 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T05:05:53.504 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:05:54.208 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T05:05:54.235 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:54.273 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:54.297 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T05:05:54.323 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T05:05:54.363 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:05:54.850 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T05:05:54.877 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T05:05:54.916 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:05:55.245 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T05:05:55.272 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T05:05:55.303 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T05:05:55.303 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:05:55.330 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T05:05:55.369 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T05:05:55.597 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T05:05:55.625 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T05:05:55.664 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T05:05:55.691 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T05:05:55.722 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:05:55.749 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:05:56.043 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T05:05:56.071 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T05:05:56.117 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:05:56.430 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T05:05:56.457 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-11T05:05:56.496 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T05:05:56.826 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T05:05:56.853 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T05:05:56.858 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T05:05:56.886 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:56.892 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T05:05:56.925 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:57.388 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T05:05:57.415 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T05:05:57.446 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:05:57.749 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T05:05:57.777 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T05:05:57.816 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T05:05:57.920 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T05:05:57.948 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T05:05:57.986 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T05:05:58.128 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T05:05:58.156 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T05:05:58.195 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:05:58.230 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T05:05:58.247 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T05:05:58.280 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:05:58.592 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T05:05:58.620 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T05:05:58.658 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:05:58.879 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T05:05:58.907 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:05:58.969 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:05:58.970 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T05:05:58.998 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-11T05:05:59.036 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T05:05:59.265 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T05:05:59.292 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T05:05:59.331 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:05:59.356 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T05:05:59.385 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T05:05:59.423 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T05:05:59.660 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T05:05:59.687 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:05:59.726 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:05:59.743 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T05:05:59.771 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T05:05:59.809 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:06:00.005 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package jq. 2024-09-11T05:06:00.021 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:06:00.063 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T05:06:00.121 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T05:06:00.148 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T05:06:00.187 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T05:06:00.283 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package socat. 2024-09-11T05:06:00.299 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T05:06:00.341 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:06:00.507 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T05:06:00.535 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T05:06:00.590 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:06:00.653 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T05:06:00.670 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T05:06:00.711 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:06:01.025 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-test. 2024-09-11T05:06:01.041 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:06:01.083 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:01.229 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T05:06:01.245 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:06:01.278 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:01.648 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T05:06:01.676 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T05:06:01.789 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:06:02.134 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T05:06:02.162 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:06:02.200 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:06:02.462 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package jq. 2024-09-11T05:06:02.490 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T05:06:02.528 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T05:06:02.731 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package socat. 2024-09-11T05:06:02.747 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T05:06:02.780 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:06:03.075 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T05:06:03.091 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T05:06:03.133 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:06:03.488 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-test. 2024-09-11T05:06:03.504 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:06:03.538 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:03.681 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T05:06:03.708 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:06:03.746 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:04.060 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T05:06:04.087 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:06:04.126 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:04.480 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T05:06:04.507 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T05:06:04.546 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:06:04.860 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T05:06:04.888 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T05:06:04.918 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:06:05.281 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T05:06:05.309 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T05:06:05.347 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:06:05.762 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package pkg-config. 2024-09-11T05:06:05.790 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T05:06:05.828 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:06:06.084 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T05:06:06.090 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T05:06:06.111 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T05:06:06.118 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T05:06:06.149 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:06.157 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:06:06.445 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T05:06:06.461 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:06:06.503 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:06.611 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T05:06:06.639 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T05:06:06.685 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T05:06:06.831 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T05:06:06.859 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T05:06:06.897 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:06:06.955 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T05:06:06.972 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T05:06:07.005 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T05:06:07.235 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T05:06:07.263 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T05:06:07.301 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T05:06:07.310 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:06:07.329 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T05:06:07.368 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:06:07.647 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T05:06:07.671 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T05:06:07.675 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T05:06:07.700 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T05:06:07.714 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:06:07.738 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T05:06:08.066 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-py. 2024-09-11T05:06:08.094 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-11T05:06:08.127 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package pkg-config. 2024-09-11T05:06:08.133 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T05:06:08.156 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T05:06:08.194 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:06:08.437 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T05:06:08.465 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T05:06:08.471 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T05:06:08.487 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T05:06:08.503 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:06:08.529 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:06:08.966 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T05:06:08.992 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T05:06:08.994 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T05:06:09.020 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T05:06:09.032 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T05:06:09.058 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:06:09.335 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T05:06:09.361 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-toml. 2024-09-11T05:06:09.363 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T05:06:09.389 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-11T05:06:09.401 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T05:06:09.419 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T05:06:09.705 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T05:06:09.715 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T05:06:09.733 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T05:06:09.742 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T05:06:09.771 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:06:09.781 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:06:10.041 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T05:06:10.069 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T05:06:10.093 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T05:06:10.107 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T05:06:10.118 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T05:06:10.151 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:06:10.413 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T05:06:10.427 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-py. 2024-09-11T05:06:10.429 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T05:06:10.455 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-11T05:06:10.488 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:06:10.493 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T05:06:10.824 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T05:06:10.851 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T05:06:10.890 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:06:11.235 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package radosgw. 2024-09-11T05:06:11.263 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../117-radosgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:06:11.301 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:11.361 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T05:06:11.389 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T05:06:11.427 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:06:11.713 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-toml. 2024-09-11T05:06:11.742 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-11T05:06:11.771 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T05:06:12.007 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T05:06:12.036 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T05:06:12.074 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:06:12.196 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T05:06:12.212 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:06:12.254 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:12.411 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T05:06:12.441 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T05:06:12.477 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:06:12.535 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T05:06:12.730 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T05:06:12.746 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T05:06:12.779 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:06:12.945 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:06:13.228 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:06:13.491 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T05:06:13.542 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package radosgw. 2024-09-11T05:06:13.570 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../117-radosgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:06:13.608 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:13.839 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T05:06:14.266 INFO:teuthology.orchestra.run.smithi138.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T05:06:14.356 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:14.511 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T05:06:14.528 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T05:06:14.569 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:14.857 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T05:06:14.881 INFO:teuthology.orchestra.run.smithi138.stdout:Adding system user cephadm....done 2024-09-11T05:06:15.046 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:06:15.099 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T05:06:15.217 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T05:06:15.298 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:06:15.458 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T05:06:15.565 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:06:15.692 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:06:15.811 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T05:06:15.943 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T05:06:16.195 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:06:16.197 INFO:teuthology.orchestra.run.smithi190.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T05:06:16.268 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:16.313 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T05:06:16.623 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:06:16.701 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user cephadm....done 2024-09-11T05:06:16.841 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T05:06:16.975 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T05:06:17.109 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T05:06:17.235 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T05:06:17.343 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T05:06:17.452 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T05:06:17.495 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:06:17.703 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T05:06:17.730 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:06:17.965 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:06:17.971 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T05:06:18.097 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T05:06:18.358 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:06:18.399 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T05:06:18.617 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:18.784 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T05:06:19.018 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T05:06:19.019 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:06:19.128 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-11T05:06:19.254 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:06:19.286 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T05:06:19.381 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:06:19.507 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:06:19.546 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T05:06:19.775 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:06:19.822 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T05:06:20.065 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T05:06:20.164 INFO:teuthology.orchestra.run.smithi138.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T05:06:20.198 INFO:teuthology.orchestra.run.smithi138.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T05:06:20.262 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:06:20.316 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:20.372 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T05:06:20.575 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T05:06:20.701 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-11T05:06:20.731 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:06:20.844 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T05:06:20.857 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T05:06:20.970 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T05:06:21.088 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T05:06:21.126 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:06:21.330 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T05:06:21.394 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:06:21.678 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T05:06:21.710 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T05:06:21.745 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T05:06:21.796 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:06:21.808 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T05:06:21.909 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T05:06:22.252 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T05:06:22.378 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T05:06:22.507 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:06:22.638 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T05:06:22.767 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:06:22.893 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:06:22.897 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T05:06:23.019 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:06:23.173 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T05:06:23.299 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T05:06:23.345 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T05:06:23.722 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T05:06:23.967 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T05:06:23.973 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:06:24.219 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T05:06:24.336 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T05:06:24.350 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:06:24.471 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T05:06:24.643 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:06:24.780 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T05:06:24.978 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:06:25.181 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T05:06:25.246 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:06:25.372 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:25.424 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T05:06:25.666 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:06:25.792 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T05:06:25.992 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:06:26.060 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T05:06:26.403 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T05:06:26.654 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T05:06:26.755 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T05:06:27.099 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:06:27.342 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:06:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T05:06:27.477 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T05:06:27.745 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:06:27.880 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:06:28.008 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:06:28.299 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:06:28.496 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T05:06:28.550 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T05:06:28.739 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:06:28.801 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:06:28.874 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T05:06:28.928 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:06:29.133 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T05:06:29.230 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:06:29.234 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T05:06:29.335 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T05:06:29.489 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T05:06:29.619 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T05:06:29.741 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T05:06:29.862 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T05:06:29.992 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:06:30.105 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:06:30.206 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T05:06:30.243 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T05:06:30.495 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:06:30.507 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T05:06:30.596 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T05:06:30.758 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T05:06:30.872 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:06:30.990 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:06:31.009 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T05:06:31.252 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T05:06:31.308 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:06:31.528 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T05:06:31.618 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:06:31.804 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T05:06:31.869 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:06:31.905 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T05:06:31.995 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:06:32.113 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:06:32.173 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T05:06:32.186 INFO:teuthology.orchestra.run.smithi138.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T05:06:32.299 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T05:06:32.432 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:06:32.558 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:06:32.935 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:06:33.025 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T05:06:33.194 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T05:06:33.320 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T05:06:33.337 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T05:06:33.572 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:06:33.585 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T05:06:33.698 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:06:33.711 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T05:06:33.846 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T05:06:33.936 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T05:06:34.000 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:06:34.181 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T05:06:34.292 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:06:34.298 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T05:06:34.568 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T05:06:34.691 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T05:06:34.735 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T05:06:34.950 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T05:06:34.955 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T05:06:35.060 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T05:06:35.300 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:06:35.336 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T05:06:35.454 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T05:06:35.568 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:06:35.763 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T05:06:35.954 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T05:06:36.047 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T05:06:36.247 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:06:36.315 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T05:06:36.441 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T05:06:36.583 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:06:36.701 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T05:06:36.710 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:36.836 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:36.962 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:06:37.045 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T05:06:37.299 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:37.330 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T05:06:37.426 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:37.552 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T05:06:37.707 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T05:06:37.828 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:37.947 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:38.000 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T05:06:38.065 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:38.274 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:06:38.274 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:06:38.327 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T05:06:38.456 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:38.579 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:38.705 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T05:06:38.766 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:38.901 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:39.027 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:39.058 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:39.153 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:06:39.159 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:39.260 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T05:06:39.528 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:39.654 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:39.780 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:39.992 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:06:39.992 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T05:06:40.391 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:40.457 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:40.500 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:40.591 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:40.609 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:40.709 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:40.735 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:40.860 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T05:06:40.861 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:41.437 INFO:teuthology.orchestra.run.smithi138.stdout:Adding group ceph....done 2024-09-11T05:06:41.814 INFO:teuthology.orchestra.run.smithi138.stdout:Adding system user ceph....done 2024-09-11T05:06:41.924 INFO:teuthology.orchestra.run.smithi138.stdout:Setting system user ceph properties....done 2024-09-11T05:06:41.942 INFO:teuthology.orchestra.run.smithi138.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T05:06:42.078 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T05:06:42.097 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:42.223 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:42.349 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:42.420 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T05:06:42.475 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:42.601 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:42.912 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:43.047 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:43.127 INFO:teuthology.orchestra.run.smithi190.stdout:Adding group ceph....done 2024-09-11T05:06:43.514 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:06:43.514 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:06:43.529 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user ceph....done 2024-09-11T05:06:43.697 INFO:teuthology.orchestra.run.smithi190.stdout:Setting system user ceph properties....done 2024-09-11T05:06:43.711 INFO:teuthology.orchestra.run.smithi190.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T05:06:43.845 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T05:06:44.152 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T05:06:44.175 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:44.525 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T05:06:44.610 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:44.728 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:44.969 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:45.160 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:06:45.160 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T05:06:45.252 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:06:45.253 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:06:45.596 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:45.687 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:45.903 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:06:45.904 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:06:45.920 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T05:06:46.372 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:46.415 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:46.638 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:06:46.639 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T05:06:46.652 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:06:46.652 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:06:47.100 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:47.157 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:47.268 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:47.400 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:47.508 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:06:47.509 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T05:06:47.622 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:06:47.622 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:06:47.988 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:48.085 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:48.236 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:48.256 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:06:48.256 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T05:06:48.362 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:48.513 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:48.731 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:48.860 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:48.912 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T05:06:49.008 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:49.224 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:06:49.224 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T05:06:49.268 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T05:06:49.710 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:49.792 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T05:06:49.877 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:49.986 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:50.113 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T05:06:50.465 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T05:06:50.533 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T05:06:50.591 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T05:06:50.735 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T05:06:51.196 INFO:teuthology.orchestra.run.smithi138.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T05:06:51.202 DEBUG:teuthology.parallel:result is None 2024-09-11T05:06:51.836 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T05:06:52.471 INFO:teuthology.orchestra.run.smithi190.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T05:06:52.475 DEBUG:teuthology.parallel:result is None 2024-09-11T05:06:52.476 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T05:06:52.624 DEBUG:teuthology.orchestra.run.smithi138:> dpkg-query -W -f '${Version}' ceph 2024-09-11T05:06:52.643 INFO:teuthology.orchestra.run.smithi138.stdout:19.3.0-4854-g8f219c47-1jammy 2024-09-11T05:06:52.644 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T05:06:52.644 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854-g8f219c47-1jammy is installed. 2024-09-11T05:06:52.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T05:06:52.787 DEBUG:teuthology.orchestra.run.smithi190:> dpkg-query -W -f '${Version}' ceph 2024-09-11T05:06:52.806 INFO:teuthology.orchestra.run.smithi190.stdout:19.3.0-4854-g8f219c47-1jammy 2024-09-11T05:06:52.807 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T05:06:52.807 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854-g8f219c47-1jammy is installed. 2024-09-11T05:06:52.809 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T05:06:52.809 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:52.809 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T05:06:52.824 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:06:52.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T05:06:52.865 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T05:06:52.865 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:52.865 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T05:06:52.885 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T05:06:52.941 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:06:52.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T05:06:52.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T05:06:53.010 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T05:06:53.010 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:53.011 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T05:06:53.024 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T05:06:53.081 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:06:53.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T05:06:53.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T05:06:53.146 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T05:06:53.147 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:53.147 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T05:06:53.160 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T05:06:53.216 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:06:53.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T05:06:53.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T05:06:53.282 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T05:06:53.377 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T05:06:53.377 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /var/log/ceph 2024-09-11T05:06:53.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-09-11T05:06:53.394 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T05:06:53.394 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T05:06:53.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T05:06:53.448 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T05:06:53.449 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T05:06:53.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T05:06:53.512 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T05:06:53.512 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T05:06:53.512 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187', 'branch': 'wip-hemanth-testing-2024-09-10-0723', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'hyelloji@redhat.com', 'first_in_suite': False, 'job_id': '7900187', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'owner': 'scheduled_hyelloji@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9167, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-hemanth-testing-2024-09-10-0723', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'targets': {'smithi138.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEm+gnqRdQvsuNDGHweFWWmcOD141W3VrgD2CVsm9QGIKs7gItQxwoWGtSYdJw1uFI+7KGxvsycXI8MKIbIxzyE=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2pNAXdrQqddEiRETthl1ihy2Tl+R5DLs9FNtzsrdOXAA7B9hNZF5TKylK4IzTnLzmuNyT4d/Sxbgp3Dk/ebrI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_17:16:14', 'tube': 'smithi', 'user': 'hyelloji', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T05:06:53.513 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T05:06:53.546 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T05:06:53.555 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T05:06:53.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T05:06:53.612 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:53.612 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T05:06:53.653 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T05:06:53.653 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2024-09-11T05:06:53.701 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T05:06:53.701 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:06:53.701 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-11T05:06:53.701 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:06:53.702 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-09-11 05:01:26.865785629 +0000 2024-09-11T05:06:53.702 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-09-11 05:01:26.637787666 +0000 2024-09-11T05:06:53.702 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-09-11 05:01:26.637787666 +0000 2024-09-11T05:06:53.702 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-09-11T05:06:53.702 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T05:06:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-09-11T05:06:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-09-11T05:06:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.00039882 s, 1.3 MB/s 2024-09-11T05:06:53.756 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T05:06:53.806 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2024-09-11T05:06:53.853 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T05:06:53.853 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:06:53.853 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-09-11T05:06:53.853 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:06:53.853 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-09-11 05:01:27.345781341 +0000 2024-09-11T05:06:53.853 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-09-11 05:01:27.089783628 +0000 2024-09-11T05:06:53.854 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-09-11 05:01:27.089783628 +0000 2024-09-11T05:06:53.854 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-09-11T05:06:53.854 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T05:06:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-09-11T05:06:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-09-11T05:06:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000441706 s, 1.2 MB/s 2024-09-11T05:06:53.908 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T05:06:53.958 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2024-09-11T05:06:54.004 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T05:06:54.005 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:06:54.005 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-11T05:06:54.005 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:06:54.005 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-09-11 05:01:27.853776803 +0000 2024-09-11T05:06:54.005 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-09-11 05:01:27.581779234 +0000 2024-09-11T05:06:54.005 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-09-11 05:01:27.581779234 +0000 2024-09-11T05:06:54.005 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-09-11T05:06:54.005 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T05:06:54.058 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-09-11T05:06:54.059 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-09-11T05:06:54.059 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000352874 s, 1.5 MB/s 2024-09-11T05:06:54.060 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T05:06:54.110 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2024-09-11T05:06:54.156 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T05:06:54.156 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:06:54.157 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-09-11T05:06:54.157 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:06:54.157 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-09-11 05:01:28.333772517 +0000 2024-09-11T05:06:54.157 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-09-11 05:01:28.113774482 +0000 2024-09-11T05:06:54.157 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-09-11 05:01:28.113774482 +0000 2024-09-11T05:06:54.157 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-09-11T05:06:54.157 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T05:06:54.211 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-09-11T05:06:54.211 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-09-11T05:06:54.211 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000381464 s, 1.3 MB/s 2024-09-11T05:06:54.212 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T05:06:54.261 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:06:54.262 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:06:54.262 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T05:06:54.268 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T05:06:54.268 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-09-11T05:06:54.316 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T05:06:54.316 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:06:54.316 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 874 Links: 1 2024-09-11T05:06:54.316 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:06:54.316 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-11 05:01:25.479949481 +0000 2024-09-11T05:06:54.317 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-11 05:01:25.475949516 +0000 2024-09-11T05:06:54.317 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-11 05:01:25.475949516 +0000 2024-09-11T05:06:54.317 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-09-11T05:06:54.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T05:06:54.371 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-11T05:06:54.371 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-11T05:06:54.371 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000448619 s, 1.1 MB/s 2024-09-11T05:06:54.372 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T05:06:54.421 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-09-11T05:06:54.469 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T05:06:54.470 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:06:54.470 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-09-11T05:06:54.470 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:06:54.470 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-11 05:01:26.199943050 +0000 2024-09-11T05:06:54.470 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-11 05:01:25.951945265 +0000 2024-09-11T05:06:54.470 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-11 05:01:25.951945265 +0000 2024-09-11T05:06:54.470 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-09-11T05:06:54.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T05:06:54.522 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-11T05:06:54.522 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-11T05:06:54.522 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000447447 s, 1.1 MB/s 2024-09-11T05:06:54.523 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T05:06:54.572 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-09-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-09-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-11 05:01:26.691938654 +0000 2024-09-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-11 05:01:26.423941049 +0000 2024-09-11T05:06:54.619 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-11 05:01:26.423941049 +0000 2024-09-11T05:06:54.620 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-09-11T05:06:54.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T05:06:54.673 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-11T05:06:54.674 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-11T05:06:54.674 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000414173 s, 1.2 MB/s 2024-09-11T05:06:54.675 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T05:06:54.725 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-09-11T05:06:54.771 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T05:06:54.771 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T05:06:54.771 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-11T05:06:54.771 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T05:06:54.771 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-11 05:01:27.159934474 +0000 2024-09-11T05:06:54.771 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-11 05:01:26.923936582 +0000 2024-09-11T05:06:54.771 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-11 05:01:26.923936582 +0000 2024-09-11T05:06:54.772 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-09-11T05:06:54.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T05:06:54.825 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-11T05:06:54.826 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-11T05:06:54.826 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000317192 s, 1.6 MB/s 2024-09-11T05:06:54.827 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T05:06:54.877 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T05:06:54.877 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-11T05:06:54.877 INFO:tasks.ceph:Generating config... 2024-09-11T05:06:54.878 INFO:tasks.ceph:[client] debug monc = 20 2024-09-11T05:06:54.878 INFO:tasks.ceph:[client] debug ms = 1 2024-09-11T05:06:54.878 INFO:tasks.ceph:[global] debug ms = 1 2024-09-11T05:06:54.878 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-11T05:06:54.878 INFO:tasks.ceph:[global] debug rados = 20 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-11T05:06:54.879 INFO:tasks.ceph:[global] ms type = async 2024-09-11T05:06:54.879 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T05:06:54.879 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-11T05:06:54.879 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T05:06:54.879 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-11T05:06:54.880 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-11T05:06:54.881 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-11T05:06:54.881 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T05:06:54.881 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-11T05:06:54.882 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-11T05:06:54.883 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T05:06:54.883 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T05:06:54.918 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T05:06:54.929 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T05:06:55.012 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T05:06:55.067 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.138'), ('mon.c', '[v2:172.21.15.138:3301,v1:172.21.15.138:6790]'), ('mon.b', '172.21.15.190')] 2024-09-11T05:06:55.068 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.138,[v2:172.21.15.138:3301,v1:172.21.15.138:6790],172.21.15.190', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-11T05:06:55.069 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:55.069 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T05:06:55.114 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.138 --addv c '[v2:172.21.15.138:3301,v1:172.21.15.138:6790]' --add b 172.21.15.190 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:06:55.200 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:06:55.200 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: generated fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:06:55.200 INFO:teuthology.orchestra.run.smithi138.stdout:setting min_mon_release = quincy 2024-09-11T05:06:55.200 INFO:teuthology.orchestra.run.smithi138.stdout:epoch 0 2024-09-11T05:06:55.200 INFO:teuthology.orchestra.run.smithi138.stdout:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:06:55.200 INFO:teuthology.orchestra.run.smithi138.stdout:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:06:55.201 INFO:teuthology.orchestra.run.smithi138.stdout:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:06:55.201 INFO:teuthology.orchestra.run.smithi138.stdout:min_mon_release 17 (quincy) 2024-09-11T05:06:55.201 INFO:teuthology.orchestra.run.smithi138.stdout:election_strategy: 1 2024-09-11T05:06:55.201 INFO:teuthology.orchestra.run.smithi138.stdout:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:06:55.201 INFO:teuthology.orchestra.run.smithi138.stdout:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:06:55.201 INFO:teuthology.orchestra.run.smithi138.stdout:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:06:55.201 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-11T05:06:55.202 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T05:06:55.249 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 66c6f018-f82d-4af3-8f0d-6530cdae2ec3... 2024-09-11T05:06:55.250 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T05:06:55.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout:[global] 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: chdir = "" 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: auth supported = cephx 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: filestore xattr use omap = true 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: mon clock drift allowed = 1.000 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: osd crush chooseleaf type = 0 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: auth debug = true 2024-09-11T05:06:55.323 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on old message = true 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on bug = true 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: mon pg warn max object skew = 0 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default size = 2 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow primary affinity = true 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow pg remap = true 2024-09-11T05:06:55.324 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on legacy crush tunables = false 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on crush straw calc version zero = false 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on no sortbitwise = false 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on osd down out interval zero = false 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on too few osds = false 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: mon_allow_pool_size_one = true 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: osd default data pool replay window = 5 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: mon allow pool delete = true 2024-09-11T05:06:55.325 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: mon cluster log file level = debug 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: debug asserts on shutdown = true 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: mon health detail to clog = false 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: mon host = "172.21.15.138,[v2:172.21.15.138:3301,v1:172.21.15.138:6790],172.21.15.190" 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: debug objecter = 20 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: debug rados = 20 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: mon client directed command retry = 5 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: mon election default strategy = 1 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: ms bind msgr1 = False 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: ms bind msgr2 = True 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: ms inject delay max = 1 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: ms inject delay probability = 0.005 2024-09-11T05:06:55.326 INFO:teuthology.orchestra.run.smithi138.stdout: ms inject delay type = mon 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: ms inject internal delays = 0.002 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: ms inject socket failures = 2500 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: ms type = async 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: fsid = 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout:[osd] 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: osd journal size = 100 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub load threshold = 5.0 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub max interval = 600 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: osd mclock profile = high_recovery_ops 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: osd recover clone overlap = true 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: osd recovery max chunk = 1048576 2024-09-11T05:06:55.327 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug shutdown = true 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug op order = true 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify stray on activate = true 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug trim objects = true 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: osd open classes on start = true 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug pg log writeout = true 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: osd deep scrub update digest min age = 30 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: osd map max advance = 10 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: journal zero on create = true 2024-09-11T05:06:55.328 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: filestore ondisk finisher threads = 3 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: filestore apply finisher threads = 3 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: bdev debug aio = true 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug misdirected ops = true 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore block size = 96636764160 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore compression algorithm = snappy 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore compression mode = aggressive 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore fsck on mount = True 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore write v2 random = True 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore zero block detection = True 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: debug bluefs = 20 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: debug bluestore = 20 2024-09-11T05:06:55.329 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: debug osd = 20 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: debug rocksdb = 10 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd full ratio = 0.9 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd nearfull ratio = 0.8 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd class default list = * 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd class load list = * 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify cached snaps = True 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify missing on start = True 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd failsafe full ratio = 0.95 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd mclock override recovery settings = True 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd objectstore = bluestore 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd op queue = debug_random 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: osd op queue cut off = debug_random 2024-09-11T05:06:55.330 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr] 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: debug mgr = 20 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: mgr/telemetry/nag = false 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: debug monc = 10 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout:[mon] 2024-09-11T05:06:55.331 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: debug paxos = 20 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: mon data avail warn = 5 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: mon mgr mkfs grace = 240 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd reporter subtree level = osd 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd prime pg temp = true 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T05:06:55.332 INFO:teuthology.orchestra.run.smithi138.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: # 1m isn't quite enough 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon_down_mkfs_grace = 2m 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_filestore_osds = false 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon min osdmap epochs = 25 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune interval = 2 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune min = 15 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune txsize = 2 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon scrub inject crc mismatch = 0.01 2024-09-11T05:06:55.333 INFO:teuthology.orchestra.run.smithi138.stdout: mon scrub inject missing keys = 0.05 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: mon scrub interval = 300 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on pool no app = False 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: paxos service trim min = 5 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout:[client] 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: rgw cache enabled = true 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable ops log = true 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable usage log = true 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: debug monc = 20 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-09-11T05:06:55.334 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.a] 2024-09-11T05:06:55.335 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.c] 2024-09-11T05:06:55.335 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.b] 2024-09-11T05:06:55.335 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-09-11T05:06:55.335 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-09-11T05:06:55.335 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T05:06:55.335 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-09-11T05:06:55.335 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.336 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T05:06:55.337 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = "172.21.15.138,[v2:172.21.15.138:3301,v1:172.21.15.138:6790],172.21.15.190" 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: debug objecter = 20 2024-09-11T05:06:55.338 INFO:teuthology.orchestra.run.smithi190.stdout: debug rados = 20 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: mon election default strategy = 1 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: ms bind msgr1 = False 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: ms bind msgr2 = True 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay max = 1 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay probability = 0.005 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject delay type = mon 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject internal delays = 0.002 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 2500 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: ms type = async 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-09-11T05:06:55.339 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-09-11T05:06:55.340 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2024-09-11T05:06:55.341 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression algorithm = snappy 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore compression mode = aggressive 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore write v2 random = True 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore zero block detection = True 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 20 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 20 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 10 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: osd class default list = * 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: osd class load list = * 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify cached snaps = True 2024-09-11T05:06:55.342 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify missing on start = True 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock override recovery settings = True 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue = debug_random 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue cut off = debug_random 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:06:55.343 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 10 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-11T05:06:55.344 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: mon min osdmap epochs = 25 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune interval = 2 2024-09-11T05:06:55.345 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune min = 15 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune txsize = 2 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub inject crc mismatch = 0.01 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub inject missing keys = 0.05 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub interval = 300 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on pool no app = False 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: paxos service trim min = 5 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2024-09-11T05:06:55.346 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T05:06:55.347 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-09-11T05:06:55.347 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.c] 2024-09-11T05:06:55.347 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.b] 2024-09-11T05:06:55.348 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T05:06:55.348 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T05:06:55.442 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T05:06:55.442 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:55.442 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T05:06:55.449 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:55.449 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T05:06:55.496 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T05:06:55.496 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:55.496 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T05:06:55.496 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T05:06:55.560 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:06:55.561 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:06:55.608 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T05:06:55.608 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:06:55.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T05:06:55.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T05:06:55.632 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:06:55.632 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:06:55.679 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T05:06:55.680 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T05:06:55.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T05:06:55.767 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T05:06:55.780 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T05:06:55.781 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T05:06:55.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T05:06:55.813 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T05:06:55.823 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T05:06:55.823 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-11T05:06:55.823 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T05:06:55.837 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:06:55.837 INFO:tasks.ceph:role: osd.0 2024-09-11T05:06:55.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T05:06:55.837 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:06:55.901 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:06:56.347 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-09-11T05:06:56.349 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-09-11T05:06:56.349 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T05:06:56.442 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T05:06:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:06:56.454 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:06:56.454 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T05:06:56.508 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:06:56.508 INFO:tasks.ceph:role: osd.1 2024-09-11T05:06:56.508 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T05:06:56.508 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T05:06:56.573 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:06:56.573 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:06:56.573 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:06:56.573 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:06:56.573 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:06:56.573 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:06:56.573 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:06:56.574 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:06:56.574 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:06:56.574 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:06:57.034 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-09-11T05:06:57.036 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-09-11T05:06:57.037 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T05:06:57.172 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T05:06:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:06:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:06:57.184 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T05:06:57.245 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T05:06:57.245 INFO:tasks.ceph:role: osd.2 2024-09-11T05:06:57.245 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T05:06:57.245 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:06:57.308 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:06:57.309 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:06:57.779 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-09-11T05:06:57.781 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-09-11T05:06:57.781 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T05:06:57.906 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T05:06:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:06:57.918 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:06:57.918 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:06:58.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:06:58.019+0000 7efeaff917c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-11T05:06:58.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:06:58.023+0000 7efeaff917c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-11T05:06:58.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:06:58.023+0000 7efeaff917c0 -1 bdev(0x556d70c87800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-11T05:06:58.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:06:58.023+0000 7efeaff917c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-11T05:07:01.165 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T05:07:01.178 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:07:01.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:01.259+0000 7f16f37e57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-11T05:07:01.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:01.259+0000 7f16f37e57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-11T05:07:01.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:01.259+0000 7f16f37e57c0 -1 bdev(0x55938ff55800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-11T05:07:01.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:01.259+0000 7f16f37e57c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-11T05:07:04.372 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T05:07:04.429 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:07:04.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:04.511+0000 7fa8dfd167c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-11T05:07:04.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:04.511+0000 7fa8dfd167c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-11T05:07:04.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:04.511+0000 7fa8dfd167c0 -1 bdev(0x55ae04ed3800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-11T05:07:04.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:04.511+0000 7fa8dfd167c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-11T05:07:07.626 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T05:07:07.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T05:07:07.698 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T05:07:07.698 INFO:tasks.ceph:role: osd.3 2024-09-11T05:07:07.699 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T05:07:07.699 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T05:07:07.762 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:07:07.762 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:07:07.762 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:07:07.762 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:07:07.762 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:07:07.762 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:07:07.762 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:07:07.762 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:07:07.763 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:07:07.763 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:07:08.217 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-11T05:07:08.219 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-11T05:07:08.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-11T05:07:08.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T05:07:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:07:08.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:07:08.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-11T05:07:08.379 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T05:07:08.380 INFO:tasks.ceph:role: osd.4 2024-09-11T05:07:08.380 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T05:07:08.380 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T05:07:08.443 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:07:08.443 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:07:08.443 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:07:08.443 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:07:08.443 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:07:08.443 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:07:08.443 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:07:08.444 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:07:08.444 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:07:08.444 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:07:08.911 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-11T05:07:08.913 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-11T05:07:08.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-11T05:07:09.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-11T05:07:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:07:09.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:07:09.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-11T05:07:09.138 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T05:07:09.138 INFO:tasks.ceph:role: osd.5 2024-09-11T05:07:09.139 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T05:07:09.139 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T05:07:09.202 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T05:07:09.203 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T05:07:09.203 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T05:07:09.203 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T05:07:09.203 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T05:07:09.203 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-11T05:07:09.203 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T05:07:09.204 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T05:07:09.204 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T05:07:09.204 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T05:07:09.684 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-11T05:07:09.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-11T05:07:09.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-11T05:07:09.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-11T05:07:09.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T05:07:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:07:09.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:07:09.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:09.945+0000 7fb30b50d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-11T05:07:09.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:09.945+0000 7fb30b50d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-11T05:07:09.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:09.945+0000 7fb30b50d7c0 -1 bdev(0x55cfae25d800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-11T05:07:09.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:09.945+0000 7fb30b50d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-11T05:07:13.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T05:07:13.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:07:13.157 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:13.153+0000 7f6fcebcd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-11T05:07:13.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:13.153+0000 7f6fcebcd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-11T05:07:13.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:13.153+0000 7f6fcebcd7c0 -1 bdev(0x55f13ebf7800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-11T05:07:13.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:13.153+0000 7f6fcebcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-11T05:07:16.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-11T05:07:16.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:07:16.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:16.437+0000 7f91a3f9f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-11T05:07:16.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:16.437+0000 7f91a3f9f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-11T05:07:16.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:16.437+0000 7f91a3f9f7c0 -1 bdev(0x5611d8515800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-11T05:07:16.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:16.437+0000 7f91a3f9f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-11T05:07:19.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T05:07:19.636 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T05:07:19.636 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:07:19.636 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T05:07:19.651 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:07:19.651 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T05:07:19.704 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:07:19.704 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T05:07:19.759 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:07:19.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T05:07:19.773 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:07:19.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T05:07:19.825 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:07:19.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-11T05:07:19.878 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:07:19.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-11T05:07:19.937 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:07:19.938 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T05:07:19.983 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T05:07:19.983 DEBUG:teuthology.orchestra.run.smithi138:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T05:07:19.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T05:07:20.028 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.0] 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQByJeFmcR5+ARAA/VkbZfzs8YGyS4wQEatujA== 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.1] 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQB1JeFmpuSVDxAABf+yS+veP8OGD+sIZ+qfTA== 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.2] 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQB4JeFm9x6bHhAAwG48PoLM63smsu0yRS3Bdw== 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr.x] 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBvJeFmiaLlLRAApouPfyO3VmzYIn6emdXMxQ== 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.3] 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQB9JeFm6Ai3OBAAO0h8RfwwenjGRH0m9PRbyA== 2024-09-11T05:07:20.029 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.4] 2024-09-11T05:07:20.030 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQCBJeFm3u+KCRAA7nbyXZ7VomKqJixaEJDIUw== 2024-09-11T05:07:20.030 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.5] 2024-09-11T05:07:20.030 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQCEJeFmoKNWGhAAj6dWWTqYFxyDaGhYMgWDVg== 2024-09-11T05:07:20.030 INFO:teuthology.orchestra.run.smithi138.stdout:[client.0] 2024-09-11T05:07:20.030 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBvJeFmMt+nMBAAtaV5NfHCoos3XnOT7q9TVw== 2024-09-11T05:07:20.036 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2024-09-11T05:07:20.036 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQByJeFmcR5+ARAA/VkbZfzs8YGyS4wQEatujA== 2024-09-11T05:07:20.036 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2024-09-11T05:07:20.036 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB1JeFmpuSVDxAABf+yS+veP8OGD+sIZ+qfTA== 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB4JeFm9x6bHhAAwG48PoLM63smsu0yRS3Bdw== 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBvJeFmiaLlLRAApouPfyO3VmzYIn6emdXMxQ== 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.3] 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB9JeFm6Ai3OBAAO0h8RfwwenjGRH0m9PRbyA== 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.4] 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCBJeFm3u+KCRAA7nbyXZ7VomKqJixaEJDIUw== 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.5] 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQCEJeFmoKNWGhAAj6dWWTqYFxyDaGhYMgWDVg== 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2024-09-11T05:07:20.037 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBvJeFmMt+nMBAAtaV5NfHCoos3XnOT7q9TVw== 2024-09-11T05:07:20.038 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.131 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.264 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T05:07:20.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T05:07:20.345 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.481 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.550 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T05:07:20.654 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:07:20.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T05:07:20.703 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T05:07:20.703 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T05:07:20.716 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T05:07:20.810 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T05:07:20.821 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-11T05:07:20.874 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T05:07:20.965 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-11T05:07:20.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-11T05:07:20.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T05:07:21.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-11T05:07:21.102 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:07:21.104 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T05:07:21.150 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T05:07:21.150 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T05:07:21.150 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T05:07:21.153 INFO:tasks.ceph.mon.a:Started 2024-09-11T05:07:21.153 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T05:07:21.153 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T05:07:21.155 INFO:tasks.ceph.mon.c:Started 2024-09-11T05:07:21.156 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T05:07:21.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T05:07:21.195 INFO:tasks.ceph.mon.b:Started 2024-09-11T05:07:21.196 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T05:07:21.196 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T05:07:21.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T05:07:21.198 INFO:tasks.ceph.mgr.x:Started 2024-09-11T05:07:21.198 DEBUG:tasks.ceph:set 0 configs 2024-09-11T05:07:21.198 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T05:07:21.198 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T05:07:21.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.311+0000 7faab3c70640 10 monclient: get_monmap_and_config 2024-09-11T05:07:21.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.311+0000 7faab3c70640 10 monclient: build_initial_monmap 2024-09-11T05:07:21.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.311+0000 7faab3c70640 10 monclient: monmap: 2024-09-11T05:07:21.315 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:21.315 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:21.315 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:21.316481+0000 2024-09-11T05:07:21.315 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:21.316481+0000 2024-09-11T05:07:21.316 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:21.316 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:21.316 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:21.316 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:21.316 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:21.316 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:21.316 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:21.316 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:21.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.311+0000 7faab3c70640 1 Processor -- start 2024-09-11T05:07:21.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 1 -- start start 2024-09-11T05:07:21.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient: init 2024-09-11T05:07:21.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:21.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-11T05:07:21.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient(hunting): picked mon.noname-c con 0x7faaac0a8350 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient(hunting): picked mon.noname-a con 0x7faaac0a6080 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faaac138900 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaac09dc70 con 0x7faaac138900 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7faaac070660 con 0x7faaac0a6080 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7faaac09d100 con 0x7faaac0a8350 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab3c70640 10 monclient(hunting): authenticate will time out at 924.752906s 2024-09-11T05:07:21.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab220d640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7faaac138900 0x7faaac138d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38558/0 (socket says 172.21.15.138:38558) 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 10 monclient(hunting): get_auth_request con 0x7faaac0a6080 auth_method 0 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab220d640 1 -- 172.21.15.138:0/2227065177 learned_addr learned my addr 172.21.15.138:0/2227065177 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faaac0a8350 auth_method 0 2024-09-11T05:07:21.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:21.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:21.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:21.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:07:21.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:21.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:21.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:07:21.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.315+0000 7faab220d640 1 -- 172.21.15.138:0/2227065177 >> v1:172.21.15.138:6789/0 conn(0x7faaac138900 legacy=0x7faaac138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 10 monclient(hunting): get_auth_request con 0x7faaac0a6080 auth_method 0 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:21.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faaac0a8350 auth_method 0 2024-09-11T05:07:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:07:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.519+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:07:21.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:21.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 10 monclient(hunting): get_auth_request con 0x7faaac0a6080 auth_method 0 2024-09-11T05:07:21.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:21.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:21.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:21.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:07:21.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faaac0a8350 auth_method 0 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:21.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:21.923+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:07:22.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:22.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 10 monclient(hunting): get_auth_request con 0x7faaac0a6080 auth_method 0 2024-09-11T05:07:22.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:22.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:22.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faaac0a8350 auth_method 0 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:22.727+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 10 monclient(hunting): tick 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 monclient(hunting): continuing hunt 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 msgr2=0x7faaac0a5b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.190:3300/0 conn(0x7faaac0a8350 0x7faaac0a5b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 msgr2=0x7faaac0a6490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 --2- 172.21.15.138:0/2227065177 >> v2:172.21.15.138:3300/0 conn(0x7faaac0a6080 0x7faaac0a6490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 >> v1:172.21.15.138:6789/0 conn(0x7faaac138900 legacy=0x7faaac138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 10 monclient: _add_conns ranks=[2,1] 2024-09-11T05:07:24.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 10 monclient(hunting): picked mon.noname-b con 0x7faa90003740 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa90006280 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaac09cb40 con 0x7faa90006280 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7faaac070660 con 0x7faa90003740 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faaa37fe640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faa90003740 auth_method 0 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:24.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:24.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.315+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:07:24.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.319+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v1:172.21.15.190:6789/0 conn(0x7faa90006280 legacy=0x7faa90008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faa90003740 auth_method 0 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:24.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.519+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:07:24.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:24.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faa90003740 auth_method 0 2024-09-11T05:07:24.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:24.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:24.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:24.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:07:24.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:24.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:24.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:24.923+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:07:25.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:25.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faa90003740 auth_method 0 2024-09-11T05:07:25.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:25.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:25.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:25.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:07:25.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:25.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:25.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:25.727+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient(hunting): tick 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 1 monclient(hunting): continuing hunt 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 >> v1:172.21.15.190:6789/0 conn(0x7faa90006280 legacy=0x7faa90008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa9000a1a0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:27.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 1 -- 172.21.15.138:0/2227065177 reap_dead start 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faaac138900 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient(hunting): picked mon.noname-b con 0x7faa90003740 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9000b0f0 con 0x7faaac138900 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa90006730 con 0x7faa9000a1a0 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 1 -- 172.21.15.138:0/2227065177 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7faaac070660 con 0x7faa90003740 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faaa37fe640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 10 monclient(hunting): get_auth_request con 0x7faa90003740 auth_method 0 2024-09-11T05:07:27.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:07:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.315+0000 7faab220d640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:07:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.319+0000 7faab120b640 1 -- 172.21.15.138:0/2227065177 >> v1:172.21.15.138:6789/0 conn(0x7faaac138900 legacy=0x7faa90010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:27.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.319+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> v1:172.21.15.190:6789/0 conn(0x7faa9000a1a0 legacy=0x7faa9000a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:27.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:27.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 10 monclient(hunting): get_auth_request con 0x7faa90003740 auth_method 0 2024-09-11T05:07:27.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:27.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:27.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:27.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:07:27.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:27.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:27.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.519+0000 7faab220d640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:07:27.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 195228934 0 0) 0x7faa9000b0f0 con 0x7faaac138900 2024-09-11T05:07:27.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): _negotiate 2024-09-11T05:07:27.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:27.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:27.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): my global_id is 4098 2024-09-11T05:07:27.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa98003530 con 0x7faaac138900 2024-09-11T05:07:27.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298934806 0 0) 0x7faa90006730 con 0x7faa9000a1a0 2024-09-11T05:07:27.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): _negotiate 2024-09-11T05:07:27.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:27.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:27.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): my global_id is 4102 2024-09-11T05:07:27.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa9000b0f0 con 0x7faa9000a1a0 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2127939941 0 0) 0x7faa98003530 con 0x7faaac138900 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa90005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa90005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 >> v1:172.21.15.190:6789/0 conn(0x7faa9000a1a0 legacy=0x7faa9000a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: _finish_hunting 0 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa90007910 con 0x7faaac138900 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: _finish_auth 0 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: _check_auth_tickets 2024-09-11T05:07:27.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7faa9c00f040 con 0x7faaac138900 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: dump: 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.771 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 >> v1:172.21.15.138:6789/0 conn(0x7faaac138900 legacy=0x7faa90010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa98003bd0 0x7faa98003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:27.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): picked mon.b con 0x7faa98003bd0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa98004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 10 monclient(hunting): picked mon.c con 0x7faa90003740 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.763+0000 7faab0a0a640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa98004e70 0x7faa9800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 10 monclient(hunting): picked mon.a con 0x7faa98004e70 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7faaac070660 con 0x7faa98004e70 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa98005630 con 0x7faa98003bd0 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7faa9800bf10 con 0x7faa90003740 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab0a0a640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab120b640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa98004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:27.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa98004e70 0x7faa9800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa98003bd0 0x7faa98003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab120b640 10 monclient(hunting): get_auth_request con 0x7faa90003740 auth_method 0 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab120b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab120b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab120b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 10 monclient(hunting): get_auth_request con 0x7faa98004e70 auth_method 0 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient(hunting): get_auth_request con 0x7faa98003bd0 auth_method 0 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:27.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab120b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab120b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab120b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 msgr2=0x7faa98004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa90003740 0x7faa98004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa98004e70 msgr2=0x7faa9800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa98004e70 0x7faa9800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient: _finish_hunting 0 2024-09-11T05:07:27.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 1 monclient: found mon.b 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa98003530 con 0x7faa98003bd0 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient: _finish_auth 0 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 10 monclient: _check_auth_tickets 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab1a0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab3c70640 5 monclient: authenticate success, global_id 4098 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab3c70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:27.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.767+0000 7faab220d640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa98004e70 0x7faa9800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:27.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab1a0c640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa98003bd0 msgr2=0x7faa98003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:27.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab1a0c640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa98003bd0 0x7faa98003fd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faaa8012a30 tx=0x7faaa801daa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62d56b19c780af7e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80ac070 con 0x7faa98003bd0 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 10 monclient: dump: 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:27.779 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa80a6840 con 0x7faa98003bd0 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 1 -- 172.21.15.138:0/2227065177 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa800f040 con 0x7faa98003bd0 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 4 set_mon_vals no callback set 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab0a0a640 10 monclient: dump: 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.780 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.781 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:27.781 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:27.781 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:27.781 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:27.781 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:27.781 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 10 monclient: shutdown 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 1 -- 172.21.15.138:0/2227065177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa98003bd0 msgr2=0x7faa98003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 1 --2- 172.21.15.138:0/2227065177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa98003bd0 0x7faa98003fd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faaa8012a30 tx=0x7faaa801daa0 comp rx=0 tx=0).stop 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab220d640 1 -- 172.21.15.138:0/2227065177 reap_dead start 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 1 -- 172.21.15.138:0/2227065177 shutdown_connections 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.775+0000 7faab3c70640 1 -- 172.21.15.138:0/2227065177 >> 172.21.15.138:0/2227065177 conn(0x7faaac0ab9f0 msgr2=0x7faaac0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 -- 172.21.15.138:0/2227065177 shutdown_connections 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 -- 172.21.15.138:0/2227065177 wait complete. 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient: build_initial_monmap 2024-09-11T05:07:27.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient: monmap: 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 librados: starting msgr at 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 librados: starting objecter 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 Processor -- start 2024-09-11T05:07:27.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 -- start start 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 librados: setting wanted keys 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 librados: calling monclient init 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient: init 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa98003bd0 0x7faaac1d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient(hunting): picked mon.noname-c con 0x7faa98003bd0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaac1d12f0 0x7faaac1262d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient(hunting): picked mon.noname-b con 0x7faaac1d12f0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faaac1d1770 0x7faaac126810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient(hunting): picked mon.noname-a con 0x7faaac1d1770 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7faaac0a6c00 con 0x7faaac1d1770 2024-09-11T05:07:27.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faaac139700 con 0x7faaac1d12f0 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7faaac09da10 con 0x7faa98003bd0 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab3c70640 10 monclient(hunting): authenticate will time out at 931.218911s 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaac1d12f0 0x7faaac1262d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaac1d12f0 0x7faaac1262d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.138:51828/0 (socket says 172.21.15.138:51828) 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 -- 172.21.15.138:0/225943335 learned_addr learned my addr 172.21.15.138:0/225943335 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient(hunting): get_auth_request con 0x7faaac1d12f0 auth_method 0 2024-09-11T05:07:27.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient(hunting): handle_auth_done global_id 4105 payload 824 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 -- 172.21.15.138:0/225943335 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa98003bd0 msgr2=0x7faaac1d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 --2- 172.21.15.138:0/225943335 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa98003bd0 0x7faaac1d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 -- 172.21.15.138:0/225943335 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faaac1d1770 msgr2=0x7faaac126810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 --2- 172.21.15.138:0/225943335 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faaac1d1770 0x7faaac126810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient: _finish_hunting 0 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 monclient: found mon.noname-b 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:27.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.190:3300/0 2024-09-11T05:07:27.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.779+0000 7faab120b640 1 -- 172.21.15.138:0/225943335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaac129850 con 0x7faaac1d12f0 2024-09-11T05:07:27.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab120b640 1 -- 172.21.15.138:0/225943335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaac1d12f0 msgr2=0x7faaac1262d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab120b640 1 --2- 172.21.15.138:0/225943335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaac1d12f0 0x7faaac1262d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faa9c015fd0 tx=0x7faa9c0146c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 client.?.objecter ms_handle_connect 0x7faaac1d12f0 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 1 -- 172.21.15.138:0/225943335 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c014e10 con 0x7faaac1d12f0 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: dump: 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:27.791 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 1 -- 172.21.15.138:0/225943335 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa9c003620 con 0x7faaac1d12f0 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 5 monclient: authenticate success, global_id 4105 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 10 monclient: _renew_subs 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaac0a2330 con 0x7faaac1d12f0 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:27.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 1 -- 172.21.15.138:0/225943335 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa9c0039d0 con 0x7faaac1d12f0 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 monclient: dump: 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:27.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 10 client.4105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 10 monclient: _renew_subs 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaac12a0f0 con 0x7faaac1d12f0 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 1 librados: init done 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa3fff640 4 set_mon_vals no callback set 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 1 -- 172.21.15.138:0/225943335 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7faa9c028060 con 0x7faaac1d12f0 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 1 -- 172.21.15.138:0/225943335 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7faa9c020740 con 0x7faaac1d12f0 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 10 client.4105.objecter ms_dispatch 0x7faaac000ba0 osd_map(1..1 src has 1..1) 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 3 client.4105.objecter handle_osd_map got epochs [1,1] > 0 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 3 client.4105.objecter handle_osd_map decoding full epoch 1 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faaa27fc640 20 client.4105.objecter dump_active .. 0 homeless 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:27.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.783+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaac13bb30 con 0x7faaac1d12f0 2024-09-11T05:07:27.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.803+0000 7faaa27fc640 1 -- 172.21.15.138:0/225943335 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7faaac129850 con 0x7faaac1d12f0 2024-09-11T05:07:27.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.803+0000 7faaa27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:27.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.803+0000 7faaa27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:27.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.875+0000 7faab3c70640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T05:07:27.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.875+0000 7faab3c70640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T05:07:27.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.875+0000 7faab3c70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:27.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:27.875+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7faaac06cbf0 con 0x7faaac1d12f0 2024-09-11T05:07:28.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faaa27fc640 1 -- 172.21.15.138:0/225943335 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7faa9c017440 con 0x7faaac1d12f0 2024-09-11T05:07:28.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faaa27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T05:07:28.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faaa27fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-11T05:07:28.130 INFO:teuthology.orchestra.run.smithi138.stderr:adjusted tunables profile to default 2024-09-11T05:07:28.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 10 librados: watch_flush enter 2024-09-11T05:07:28.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 10 librados: watch_flush exit 2024-09-11T05:07:28.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 20 client.4105.objecter shutdown clearing up homeless session... 2024-09-11T05:07:28.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 10 client.4105.objecter successfully canceled tick 2024-09-11T05:07:28.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 10 monclient: shutdown 2024-09-11T05:07:28.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaac1d12f0 msgr2=0x7faaac1262d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 --2- 172.21.15.138:0/225943335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaac1d12f0 0x7faaac1262d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faa9c015fd0 tx=0x7faa9c0146c0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 shutdown_connections 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 --2- 172.21.15.138:0/225943335 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faaac1d1770 0x7faaac126810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 --2- 172.21.15.138:0/225943335 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faaac1d12f0 0x7faaac1262d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 --2- 172.21.15.138:0/225943335 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa98003bd0 0x7faaac1d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 >> 172.21.15.138:0/225943335 conn(0x7faaac0ab9f0 msgr2=0x7faaac0a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 shutdown_connections 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 -- 172.21.15.138:0/225943335 wait complete. 2024-09-11T05:07:28.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:28.127+0000 7faab3c70640 1 librados: shutdown 2024-09-11T05:07:28.152 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T05:07:28.152 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T05:07:28.152 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:07:28.153 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T05:07:28.164 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:07:28.164 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T05:07:28.218 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-11T05:07:28.218 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T05:07:28.274 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:07:28.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T05:07:28.286 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:07:28.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-11T05:07:28.341 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:07:28.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-11T05:07:28.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 79670fd2-b5aa-408e-8aa4-18a8fb848972 0 2024-09-11T05:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient: get_monmap_and_config 2024-09-11T05:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient: build_initial_monmap 2024-09-11T05:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient: monmap: 2024-09-11T05:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:07:28.567238+0000 2024-09-11T05:07:28.564 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:07:28.567238+0000 2024-09-11T05:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:28.565 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 1 Processor -- start 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 1 -- start start 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient: init 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43280a3a20 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a4510 0x7f432809d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient(hunting): picked mon.noname-b con 0x7f43280a4510 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f432813da40 0x7f432813fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient(hunting): picked mon.noname-c con 0x7f432813da40 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4328074aa0 con 0x7f43280a3a20 2024-09-11T05:07:28.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f432809cf00 con 0x7f432813da40 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f432809cc20 con 0x7f43280a4510 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432f8f5640 10 monclient(hunting): authenticate will time out at 929.359404s 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432d691640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f43280a3a20 0x7f43280a3e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:53018/0 (socket says 172.21.15.190:53018) 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f432813da40 0x7f432813fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432d691640 1 -- 172.21.15.190:0/3615027802 learned_addr learned my addr 172.21.15.190:0/3615027802 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient(hunting): get_auth_request con 0x7f432813da40 auth_method 0 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:28.569 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432ce90640 1 --2- 172.21.15.190:0/3615027802 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a4510 0x7f432809d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 1 -- 172.21.15.190:0/3615027802 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a4510 msgr2=0x7f432809d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 1 --2- 172.21.15.190:0/3615027802 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a4510 0x7f432809d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 1 -- 172.21.15.190:0/3615027802 >> v1:172.21.15.190:6789/0 conn(0x7f43280a3a20 legacy=0x7f43280a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient: _finish_hunting 0 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 1 monclient: found mon.noname-c 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.570 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.561+0000 7f432de92640 1 -- 172.21.15.190:0/3615027802 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432809d1e0 con 0x7f432813da40 2024-09-11T05:07:28.574 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432de92640 1 -- 172.21.15.190:0/3615027802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f432813da40 msgr2=0x7f432813fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:28.574 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432de92640 1 --2- 172.21.15.190:0/3615027802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f432813da40 0x7f432813fe30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4320009d90 tx=0x7f432001c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2524ccf091d651 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 1 -- 172.21.15.190:0/3615027802 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200b2070 con 0x7f432813da40 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: dump: 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: _finish_auth 0 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 1 -- 172.21.15.190:0/3615027802 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43200a5870 con 0x7f432813da40 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 1 -- 172.21.15.190:0/3615027802 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200a5c40 con 0x7f432813da40 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 5 monclient: authenticate success, global_id 4111 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 4 set_mon_vals no callback set 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f431ffff640 10 monclient: dump: 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.576 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient: shutdown 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- 172.21.15.190:0/3615027802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f432813da40 msgr2=0x7f432813fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 --2- 172.21.15.190:0/3615027802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f432813da40 0x7f432813fe30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4320009d90 tx=0x7f432001c830 comp rx=0 tx=0).stop 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- 172.21.15.190:0/3615027802 shutdown_connections 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 --2- 172.21.15.190:0/3615027802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f432813da40 0x7f432813fe30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 --2- 172.21.15.190:0/3615027802 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a4510 0x7f432809d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- 172.21.15.190:0/3615027802 >> 172.21.15.190:0/3615027802 conn(0x7f43280a52c0 msgr2=0x7f4328076910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- 172.21.15.190:0/3615027802 shutdown_connections 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- 172.21.15.190:0/3615027802 wait complete. 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient: build_initial_monmap 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient: monmap: 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:28.578 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 librados: starting msgr at 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 librados: starting objecter 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 Processor -- start 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- start start 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 librados: setting wanted keys 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 librados: calling monclient init 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient: init 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a3a20 0x7f4328121a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient(hunting): picked mon.noname-c con 0x7f43280a3a20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43280a4510 0x7f43281388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient(hunting): picked mon.noname-b con 0x7f43280a4510 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f432813da40 0x7f43281d0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient(hunting): picked mon.noname-a con 0x7f432813da40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4328142f20 con 0x7f432813da40 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4328142da0 con 0x7f43280a4510 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f43281430a0 con 0x7f43280a3a20 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432f8f5640 10 monclient(hunting): authenticate will time out at 929.366605s 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43280a4510 0x7f43281388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43280a4510 0x7f43281388f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:34838/0 (socket says 172.21.15.190:34838) 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 -- 172.21.15.190:0/2162406341 learned_addr learned my addr 172.21.15.190:0/2162406341 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient(hunting): get_auth_request con 0x7f43280a4510 auth_method 0 2024-09-11T05:07:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432de92640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f432813da40 0x7f43281d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432d691640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a3a20 0x7f4328121a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 -- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a3a20 msgr2=0x7f4328121a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a3a20 0x7f4328121a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 -- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f432813da40 msgr2=0x7f43281d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f432813da40 0x7f43281d0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient: _finish_hunting 0 2024-09-11T05:07:28.581 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 monclient: found mon.noname-b 2024-09-11T05:07:28.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:28.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432ce90640 1 -- 172.21.15.190:0/2162406341 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4328122fa0 con 0x7f43280a4510 2024-09-11T05:07:28.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432d691640 10 monclient: get_auth_request con 0x7f43280a3a20 auth_method 0 2024-09-11T05:07:28.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432d691640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a3a20 0x7f4328121a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:28.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432de92640 10 monclient: get_auth_request con 0x7f432813da40 auth_method 0 2024-09-11T05:07:28.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.569+0000 7f432de92640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f432813da40 0x7f43281d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432ce90640 1 -- 172.21.15.190:0/2162406341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43280a4510 msgr2=0x7f43281388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432ce90640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43280a4510 0x7f43281388f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f431800c7f0 tx=0x7f431800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 client.?.objecter ms_handle_connect 0x7f43280a4510 2024-09-11T05:07:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 1 -- 172.21.15.190:0/2162406341 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4318015070 con 0x7f43280a4510 2024-09-11T05:07:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: dump: 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:28.585 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 1 -- 172.21.15.190:0/2162406341 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4318004220 con 0x7f43280a4510 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 5 monclient: authenticate success, global_id 4114 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 10 monclient: _renew_subs 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4328123280 con 0x7f43280a4510 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 1 -- 172.21.15.190:0/2162406341 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4318007830 con 0x7f43280a4510 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 monclient: dump: 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:28.586 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431effd640 4 set_mon_vals no callback set 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 10 monclient: _renew_subs 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4328120f20 con 0x7f43280a4510 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 1 librados: init done 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 1 -- 172.21.15.190:0/2162406341 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f43180079d0 con 0x7f43280a4510 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 1 -- 172.21.15.190:0/2162406341 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f43180194d0 con 0x7f43280a4510 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 10 client.4114.objecter ms_dispatch 0x7f4328000ba0 osd_map(2..2 src has 1..2) 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-09-11T05:07:28.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f431dffb640 20 client.4114.objecter dump_active .. 0 homeless 2024-09-11T05:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.573+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f432809d8e0 con 0x7f43280a4510 2024-09-11T05:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.577+0000 7f431dffb640 1 -- 172.21.15.190:0/2162406341 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f4318003620 con 0x7f43280a4510 2024-09-11T05:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.577+0000 7f431dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:28.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.577+0000 7f431dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:28.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.649+0000 7f432f8f5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "79670fd2-b5aa-408e-8aa4-18a8fb848972", "id": 0}] 2024-09-11T05:07:28.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.649+0000 7f432f8f5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "79670fd2-b5aa-408e-8aa4-18a8fb848972", "id": 0}] 2024-09-11T05:07:28.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.649+0000 7f432f8f5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.649+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "79670fd2-b5aa-408e-8aa4-18a8fb848972", "id": 0} v 0) -- 0x7f4328122b30 con 0x7f43280a4510 2024-09-11T05:07:28.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f431dffb640 1 -- 172.21.15.190:0/2162406341 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "79670fd2-b5aa-408e-8aa4-18a8fb848972", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f4318003800 con 0x7f43280a4510 2024-09-11T05:07:28.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f431dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "79670fd2-b5aa-408e-8aa4-18a8fb848972", "id": 0}] 2024-09-11T05:07:28.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f431dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:28.657 INFO:teuthology.orchestra.run.smithi190.stdout:0 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 10 librados: watch_flush enter 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 10 librados: watch_flush exit 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 20 client.4114.objecter shutdown clearing up homeless session... 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 10 client.4114.objecter successfully canceled tick 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 10 monclient: shutdown 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43280a4510 msgr2=0x7f43281388f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43280a4510 0x7f43281388f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f431800c7f0 tx=0x7f431800ccb0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 shutdown_connections 2024-09-11T05:07:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f432813da40 0x7f43281d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43280a4510 0x7f43281388f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 --2- 172.21.15.190:0/2162406341 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43280a3a20 0x7f4328121a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 >> 172.21.15.190:0/2162406341 conn(0x7f43280a52c0 msgr2=0x7f43280a1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:28.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 shutdown_connections 2024-09-11T05:07:28.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.653+0000 7f432f8f5640 1 -- 172.21.15.190:0/2162406341 wait complete. 2024-09-11T05:07:28.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.657+0000 7f432f8f5640 1 librados: shutdown 2024-09-11T05:07:28.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new e96fe3a6-3058-4c40-aa87-26a89572cdfd 1 2024-09-11T05:07:28.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.777+0000 7f61b9a2f640 10 monclient: get_monmap_and_config 2024-09-11T05:07:28.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.777+0000 7f61b9a2f640 10 monclient: build_initial_monmap 2024-09-11T05:07:28.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.777+0000 7f61b9a2f640 10 monclient: monmap: 2024-09-11T05:07:28.781 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:07:28.784700+0000 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:07:28.784700+0000 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:28.782 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 1 Processor -- start 2024-09-11T05:07:28.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 1 -- start start 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient: init 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a48e0 0x7f61b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient(hunting): picked mon.noname-b con 0x7f61b40a48e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61b406fc80 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61b40a5200 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b406eed0 con 0x7f61b406fc80 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b40703f0 con 0x7f61b40a5200 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f61b40a3430 con 0x7f61b40a48e0 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b9a2f640 10 monclient(hunting): authenticate will time out at 929.576927s 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b37fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f61b40a5200 0x7f61b41401d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:53028/0 (socket says 172.21.15.190:53028) 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b37fe640 1 -- 172.21.15.190:0/4193560271 learned_addr learned my addr 172.21.15.190:0/4193560271 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 1 --2- 172.21.15.190:0/4193560271 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a48e0 0x7f61b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient(hunting): get_auth_request con 0x7f61b40a48e0 auth_method 0 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient(hunting): handle_auth_done global_id 4100 payload 274 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 1 -- 172.21.15.190:0/4193560271 >> v1:172.21.15.190:6789/0 conn(0x7f61b40a5200 legacy=0x7f61b41401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 1 -- 172.21.15.190:0/4193560271 >> v1:172.21.15.138:6789/0 conn(0x7f61b406fc80 legacy=0x7f61b413caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 1 monclient: found mon.noname-b 2024-09-11T05:07:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.781+0000 7f61b2ffd640 1 -- 172.21.15.190:0/4193560271 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b406f3e0 con 0x7f61b40a48e0 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b2ffd640 1 -- 172.21.15.190:0/4193560271 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a48e0 msgr2=0x7f61b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b2ffd640 1 --2- 172.21.15.190:0/4193560271 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a48e0 0x7f61b40a4cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f61a40097f0 tx=0x7f61a401c190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f5d513211871465 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 1 -- 172.21.15.190:0/4193560271 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a40b0070 con 0x7f61b40a48e0 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: dump: 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 5 monclient: authenticate success, global_id 4100 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 1 -- 172.21.15.190:0/4193560271 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a40ab600 con 0x7f61b40a48e0 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 1 -- 172.21.15.190:0/4193560271 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a40ab9d0 con 0x7f61b40a48e0 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b1ffb640 10 monclient: dump: 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 4 set_mon_vals no callback set 2024-09-11T05:07:28.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 10 monclient: shutdown 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 1 -- 172.21.15.190:0/4193560271 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a48e0 msgr2=0x7f61b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 1 --2- 172.21.15.190:0/4193560271 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a48e0 0x7f61b40a4cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f61a40097f0 tx=0x7f61a401c190 comp rx=0 tx=0).stop 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 1 -- 172.21.15.190:0/4193560271 shutdown_connections 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 1 --2- 172.21.15.190:0/4193560271 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a48e0 0x7f61b40a4cc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 1 -- 172.21.15.190:0/4193560271 >> 172.21.15.190:0/4193560271 conn(0x7f61b40a94d0 msgr2=0x7f61b40a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 1 -- 172.21.15.190:0/4193560271 shutdown_connections 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.785+0000 7f61b9a2f640 1 -- 172.21.15.190:0/4193560271 wait complete. 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient: build_initial_monmap 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient: monmap: 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T05:07:28.793 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T05:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 librados: starting msgr at 2024-09-11T05:07:28.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 librados: starting objecter 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 Processor -- start 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 -- start start 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 librados: setting wanted keys 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 librados: calling monclient init 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient: init 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b406fc80 0x7f61b4128b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient(hunting): picked mon.noname-b con 0x7f61b406fc80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f61b40a48e0 0x7f61b4127300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient(hunting): picked mon.noname-a con 0x7f61b40a48e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a5200 0x7f61b41d0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient(hunting): picked mon.noname-c con 0x7f61b40a5200 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f61b40a64c0 con 0x7f61b40a48e0 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f61b40a6340 con 0x7f61b406fc80 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f61b4144520 con 0x7f61b40a5200 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient(hunting): authenticate will time out at 929.585049s 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b406fc80 0x7f61b4128b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.800 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b406fc80 0x7f61b4128b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:34854/0 (socket says 172.21.15.190:34854) 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 -- 172.21.15.190:0/1483534473 learned_addr learned my addr 172.21.15.190:0/1483534473 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient(hunting): get_auth_request con 0x7f61b406fc80 auth_method 0 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b27fc640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f61b40a48e0 0x7f61b4127300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b37fe640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a5200 0x7f61b41d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 -- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a5200 msgr2=0x7f61b41d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a5200 0x7f61b41d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 -- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f61b40a48e0 msgr2=0x7f61b4127300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f61b40a48e0 0x7f61b4127300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 monclient: found mon.noname-b 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 -- 172.21.15.190:0/1483534473 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b412a060 con 0x7f61b406fc80 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b27fc640 10 monclient: get_auth_request con 0x7f61b40a48e0 auth_method 0 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b27fc640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f61b40a48e0 0x7f61b4127300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 -- 172.21.15.190:0/1483534473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b406fc80 msgr2=0x7f61b4128b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b2ffd640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b406fc80 0x7f61b4128b40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61a40ae040 tx=0x7f61a401a9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 10 client.?.objecter ms_handle_connect 0x7f61b406fc80 2024-09-11T05:07:28.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 1 -- 172.21.15.190:0/1483534473 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a40b0050 con 0x7f61b406fc80 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 10 monclient: dump: 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 10 monclient: _finish_auth 0 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 5 monclient: authenticate success, global_id 4117 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient: _renew_subs 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f619bfff640 1 -- 172.21.15.190:0/1483534473 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a40bc070 con 0x7f61b406fc80 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.789+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b412a340 con 0x7f61b406fc80 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 1 -- 172.21.15.190:0/1483534473 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a40b5c80 con 0x7f61b406fc80 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:28.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 10 monclient: dump: 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 10 monclient: _renew_subs 2024-09-11T05:07:28.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b17fa640 4 set_mon_vals no callback set 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b4127fe0 con 0x7f61b406fc80 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 1 librados: init done 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 1 -- 172.21.15.190:0/1483534473 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f61a40c1460 con 0x7f61b406fc80 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 1 -- 172.21.15.190:0/1483534473 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f61a40b7800 con 0x7f61b406fc80 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 10 client.4117.objecter ms_dispatch 0x7f61b4000ba0 osd_map(3..3 src has 1..3) 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f619bfff640 20 client.4117.objecter dump_active .. 0 homeless 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.793+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b4129c20 con 0x7f61b406fc80 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.797+0000 7f619bfff640 1 -- 172.21.15.190:0/1483534473 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f61b40703f0 con 0x7f61b406fc80 2024-09-11T05:07:28.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.797+0000 7f619bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:28.807 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.797+0000 7f619bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:28.869 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.865+0000 7f61b9a2f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e96fe3a6-3058-4c40-aa87-26a89572cdfd", "id": 1}] 2024-09-11T05:07:28.869 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.865+0000 7f61b9a2f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e96fe3a6-3058-4c40-aa87-26a89572cdfd", "id": 1}] 2024-09-11T05:07:28.869 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.865+0000 7f61b9a2f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:28.869 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.865+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e96fe3a6-3058-4c40-aa87-26a89572cdfd", "id": 1} v 0) -- 0x7f61b4141020 con 0x7f61b406fc80 2024-09-11T05:07:28.877 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.873+0000 7f619bfff640 1 -- 172.21.15.190:0/1483534473 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e96fe3a6-3058-4c40-aa87-26a89572cdfd", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f61a40ab5a0 con 0x7f61b406fc80 2024-09-11T05:07:28.877 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.873+0000 7f619bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e96fe3a6-3058-4c40-aa87-26a89572cdfd", "id": 1}] 2024-09-11T05:07:28.877 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.873+0000 7f619bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:28.877 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-09-11T05:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 10 librados: watch_flush enter 2024-09-11T05:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 10 librados: watch_flush exit 2024-09-11T05:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 20 client.4117.objecter shutdown clearing up homeless session... 2024-09-11T05:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 10 client.4117.objecter successfully canceled tick 2024-09-11T05:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 10 monclient: shutdown 2024-09-11T05:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:28.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b406fc80 msgr2=0x7f61b4128b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b406fc80 0x7f61b4128b40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61a40ae040 tx=0x7f61a401a9d0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 shutdown_connections 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f61b40a5200 0x7f61b41d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f61b40a48e0 0x7f61b4127300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 --2- 172.21.15.190:0/1483534473 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f61b406fc80 0x7f61b4128b40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 >> 172.21.15.190:0/1483534473 conn(0x7f61b40a94d0 msgr2=0x7f61b409ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 shutdown_connections 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 -- 172.21.15.190:0/1483534473 wait complete. 2024-09-11T05:07:28.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:28.877+0000 7f61b9a2f640 1 librados: shutdown 2024-09-11T05:07:28.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new a87dbce5-b78c-4070-90cc-d323fc444f4b 2 2024-09-11T05:07:29.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.005+0000 7fa4fb7c2640 10 monclient: get_monmap_and_config 2024-09-11T05:07:29.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.005+0000 7fa4fb7c2640 10 monclient: build_initial_monmap 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.005+0000 7fa4fb7c2640 10 monclient: monmap: 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:07:29.012644+0000 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:07:29.012644+0000 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:29.010 INFO:teuthology.orchestra.run.smithi190.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:29.011 INFO:teuthology.orchestra.run.smithi190.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:29.011 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 1 Processor -- start 2024-09-11T05:07:29.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 1 -- start start 2024-09-11T05:07:29.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient: init 2024-09-11T05:07:29.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:29.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-11T05:07:29.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa4f40a2d90 0x7fa4f409f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f40a2d90 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4f409fed0 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4f40a02b0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4f40a16c0 con 0x7fa4f409fed0 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4f409e390 con 0x7fa4f40a02b0 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fa4f406ec20 con 0x7fa4f40a2d90 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4fb7c2640 10 monclient(hunting): authenticate will time out at 929.804688s 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f9d5f640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa4f40a02b0 0x7fa4f409a770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:53044/0 (socket says 172.21.15.190:53044) 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f9d5f640 1 -- 172.21.15.190:0/2374869954 learned_addr learned my addr 172.21.15.190:0/2374869954 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-09-11T05:07:29.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 1 --2- 172.21.15.190:0/2374869954 >> v2:172.21.15.138:3300/0 conn(0x7fa4f40a2d90 0x7fa4f409f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient(hunting): get_auth_request con 0x7fa4f40a2d90 auth_method 0 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 1 -- 172.21.15.190:0/2374869954 >> v1:172.21.15.190:6789/0 conn(0x7fa4f40a02b0 legacy=0x7fa4f409a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 1 -- 172.21.15.190:0/2374869954 >> v1:172.21.15.138:6789/0 conn(0x7fa4f409fed0 legacy=0x7fa4f409a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient: _finish_hunting 0 2024-09-11T05:07:29.015 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 1 monclient: found mon.noname-a 2024-09-11T05:07:29.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:29.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:29.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.009+0000 7fa4f955e640 1 -- 172.21.15.190:0/2374869954 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f40a0b40 con 0x7fa4f40a2d90 2024-09-11T05:07:29.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4f955e640 1 -- 172.21.15.190:0/2374869954 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f40a2d90 msgr2=0x7fa4f409f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:29.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4f955e640 1 --2- 172.21.15.190:0/2374869954 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409f990 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00097c0 tx=0x7fa4f001c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198b40f8a4808ed5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:29.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 1 -- 172.21.15.190:0/2374869954 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f00b0070 con 0x7fa4f40a2d90 2024-09-11T05:07:29.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:29.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:29.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: dump: 2024-09-11T05:07:29.020 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: _finish_auth 0 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:29.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 1 -- 172.21.15.190:0/2374869954 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f00ab600 con 0x7fa4f40a2d90 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 5 monclient: authenticate success, global_id 4107 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 1 -- 172.21.15.190:0/2374869954 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f00ab9d0 con 0x7fa4f40a2d90 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 4 set_mon_vals no callback set 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4ebfff640 10 monclient: dump: 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:29.022 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 10 monclient: shutdown 2024-09-11T05:07:29.023 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/2374869954 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f40a2d90 msgr2=0x7fa4f409f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 --2- 172.21.15.190:0/2374869954 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409f990 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00097c0 tx=0x7fa4f001c180 comp rx=0 tx=0).stop 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/2374869954 shutdown_connections 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 --2- 172.21.15.190:0/2374869954 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409f990 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/2374869954 >> 172.21.15.190:0/2374869954 conn(0x7fa4f40a3e80 msgr2=0x7fa4f40a4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/2374869954 shutdown_connections 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/2374869954 wait complete. 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 10 monclient: build_initial_monmap 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 10 monclient: monmap: 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:29.024 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 librados: starting msgr at 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 librados: starting objecter 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 Processor -- start 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 -- start start 2024-09-11T05:07:29.025 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 librados: setting wanted keys 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 1 librados: calling monclient init 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.013+0000 7fa4fb7c2640 10 monclient: init 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f409fed0 0x7fa4f4135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f409fed0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4f40a02b0 0x7fa4f409a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f40a02b0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f40a2d90 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f40a0e20 con 0x7fa4f409fed0 2024-09-11T05:07:29.026 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f40a3170 con 0x7fa4f40a2d90 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f409ae80 con 0x7fa4f40a02b0 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient(hunting): authenticate will time out at 929.812408s 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:34860/0 (socket says 172.21.15.190:34860) 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 -- 172.21.15.190:0/445432179 learned_addr learned my addr 172.21.15.190:0/445432179 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient(hunting): get_auth_request con 0x7fa4f40a2d90 auth_method 0 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:29.027 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f8d5d640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4f40a02b0 0x7fa4f409a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f955e640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f409fed0 0x7fa4f4135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f8d5d640 10 monclient(hunting): get_auth_request con 0x7fa4f40a02b0 auth_method 0 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f8d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f8d5d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f8d5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 -- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4f40a02b0 msgr2=0x7fa4f409a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:29.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4f40a02b0 0x7fa4f409a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 -- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f409fed0 msgr2=0x7fa4f4135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f409fed0 0x7fa4f4135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient: _finish_hunting 0 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 monclient: found mon.noname-b 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 -- 172.21.15.190:0/445432179 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f413d270 con 0x7fa4f40a2d90 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f955e640 10 monclient: get_auth_request con 0x7fa4f409fed0 auth_method 0 2024-09-11T05:07:29.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f955e640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f409fed0 0x7fa4f4135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:29.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 -- 172.21.15.190:0/445432179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f40a2d90 msgr2=0x7fa4f409a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:29.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 client.?.objecter ms_handle_connect 0x7fa4f40a2d90 2024-09-11T05:07:29.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4f9d5f640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409a5a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00caf0 tx=0x7fa4ec00a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 1 -- 172.21.15.190:0/445432179 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec012400 con 0x7fa4f40a2d90 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: dump: 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 1 -- 172.21.15.190:0/445432179 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4ec0125c0 con 0x7fa4f40a2d90 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 5 monclient: authenticate success, global_id 4120 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient: _renew_subs 2024-09-11T05:07:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f413d550 con 0x7fa4f40a2d90 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 1 -- 172.21.15.190:0/445432179 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0129b0 con 0x7fa4f40a2d90 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 monclient: dump: 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4eb7fe640 4 set_mon_vals no callback set 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient: _renew_subs 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f409bb80 con 0x7fa4f40a2d90 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4fb7c2640 1 librados: init done 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 1 -- 172.21.15.190:0/445432179 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fa4ec012bd0 con 0x7fa4f40a2d90 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 1 -- 172.21.15.190:0/445432179 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fa4ec0074d0 con 0x7fa4f40a2d90 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 10 client.4120.objecter ms_dispatch 0x7fa4f4000ba0 osd_map(4..4 src has 1..4) 2024-09-11T05:07:29.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 3 client.4120.objecter handle_osd_map decoding full epoch 4 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.017+0000 7fa4e9ffb640 20 client.4120.objecter dump_active .. 0 homeless 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.021+0000 7fa4fb7c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.021+0000 7fa4fb7c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.021+0000 7fa4fb7c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.021+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f409f990 con 0x7fa4f40a2d90 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.021+0000 7fa4e9ffb640 1 -- 172.21.15.190:0/445432179 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fa4ec007070 con 0x7fa4f40a2d90 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.021+0000 7fa4e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:29.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.021+0000 7fa4e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:29.095 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.093+0000 7fa4fb7c2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a87dbce5-b78c-4070-90cc-d323fc444f4b", "id": 2}] 2024-09-11T05:07:29.095 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.093+0000 7fa4fb7c2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a87dbce5-b78c-4070-90cc-d323fc444f4b", "id": 2}] 2024-09-11T05:07:29.095 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.093+0000 7fa4fb7c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.095 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.093+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a87dbce5-b78c-4070-90cc-d323fc444f4b", "id": 2} v 0) -- 0x7fa4f413e1b0 con 0x7fa4f40a2d90 2024-09-11T05:07:29.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.301+0000 7fa4e9ffb640 1 -- 172.21.15.190:0/445432179 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a87dbce5-b78c-4070-90cc-d323fc444f4b", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fa4ec0077f0 con 0x7fa4f40a2d90 2024-09-11T05:07:29.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a87dbce5-b78c-4070-90cc-d323fc444f4b", "id": 2}] 2024-09-11T05:07:29.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:29.306 INFO:teuthology.orchestra.run.smithi190.stdout:2 2024-09-11T05:07:29.308 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 10 librados: watch_flush enter 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 10 librados: watch_flush exit 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 20 client.4120.objecter shutdown clearing up homeless session... 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 10 client.4120.objecter successfully canceled tick 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 10 monclient: shutdown 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f40a2d90 msgr2=0x7fa4f409a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409a5a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00caf0 tx=0x7fa4ec00a060 comp rx=0 tx=0).stop 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 shutdown_connections 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4f40a2d90 0x7fa4f409a5a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4f40a02b0 0x7fa4f409a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 --2- 172.21.15.190:0/445432179 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4f409fed0 0x7fa4f4135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 >> 172.21.15.190:0/445432179 conn(0x7fa4f40a3e80 msgr2=0x7fa4f409e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 shutdown_connections 2024-09-11T05:07:29.310 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 -- 172.21.15.190:0/445432179 wait complete. 2024-09-11T05:07:29.310 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.305+0000 7fa4fb7c2640 1 librados: shutdown 2024-09-11T05:07:29.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new e1a7ea3f-62f5-4d1f-b671-be183143eb20 3 2024-09-11T05:07:29.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient: get_monmap_and_config 2024-09-11T05:07:29.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient: build_initial_monmap 2024-09-11T05:07:29.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient: monmap: 2024-09-11T05:07:29.444 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:07:29.447723+0000 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:07:29.447723+0000 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:29.445 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 1 Processor -- start 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 1 -- start start 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient: init 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100748d0 0x7f7610138940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient(hunting): picked mon.noname-b con 0x7f76100748d0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f761006d3a0 0x7f7610138e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient(hunting): picked mon.noname-a con 0x7f761006d3a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76100a3c30 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f761009f190 con 0x7f76100a3c30 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f7610074cb0 con 0x7f761006d3a0 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f761009cc50 con 0x7f76100748d0 2024-09-11T05:07:29.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f7615ffc640 10 monclient(hunting): authenticate will time out at 930.239824s 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f760ffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f76100a3c30 0x7f76100a40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:53048/0 (socket says 172.21.15.190:53048) 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.441+0000 7f760ffff640 1 -- 172.21.15.190:0/4120637343 learned_addr learned my addr 172.21.15.190:0/4120637343 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 1 --2- 172.21.15.190:0/4120637343 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100748d0 0x7f7610138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient(hunting): get_auth_request con 0x7f76100748d0 auth_method 0 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760effd640 1 --2- 172.21.15.190:0/4120637343 >> v2:172.21.15.138:3300/0 conn(0x7f761006d3a0 0x7f7610138e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760effd640 10 monclient(hunting): get_auth_request con 0x7f761006d3a0 auth_method 0 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 1 -- 172.21.15.190:0/4120637343 >> v2:172.21.15.138:3300/0 conn(0x7f761006d3a0 msgr2=0x7f7610138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 1 --2- 172.21.15.190:0/4120637343 >> v2:172.21.15.138:3300/0 conn(0x7f761006d3a0 0x7f7610138e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 1 -- 172.21.15.190:0/4120637343 >> v1:172.21.15.190:6789/0 conn(0x7f76100a3c30 legacy=0x7f76100a40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 1 monclient: found mon.noname-b 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:29.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.445+0000 7f760f7fe640 1 -- 172.21.15.190:0/4120637343 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761009cf30 con 0x7f76100748d0 2024-09-11T05:07:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760f7fe640 1 -- 172.21.15.190:0/4120637343 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100748d0 msgr2=0x7f7610138940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760f7fe640 1 --2- 172.21.15.190:0/4120637343 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100748d0 0x7f7610138940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7610074fb0 tx=0x7f760801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b288bed3953027f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 1 -- 172.21.15.190:0/4120637343 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76080ac070 con 0x7f76100748d0 2024-09-11T05:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: dump: 2024-09-11T05:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.452 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 1 -- 172.21.15.190:0/4120637343 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76080a6640 con 0x7f76100748d0 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 5 monclient: authenticate success, global_id 4106 2024-09-11T05:07:29.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 1 -- 172.21.15.190:0/4120637343 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76080a6a10 con 0x7f76100748d0 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 4 set_mon_vals no callback set 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f760e7fc640 10 monclient: dump: 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.454 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient: shutdown 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- 172.21.15.190:0/4120637343 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100748d0 msgr2=0x7f7610138940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 --2- 172.21.15.190:0/4120637343 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100748d0 0x7f7610138940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7610074fb0 tx=0x7f760801c7f0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- 172.21.15.190:0/4120637343 shutdown_connections 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 --2- 172.21.15.190:0/4120637343 >> v2:172.21.15.138:3300/0 conn(0x7f761006d3a0 0x7f7610138e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 --2- 172.21.15.190:0/4120637343 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100748d0 0x7f7610138940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- 172.21.15.190:0/4120637343 >> 172.21.15.190:0/4120637343 conn(0x7f76100a55d0 msgr2=0x7f761006ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- 172.21.15.190:0/4120637343 shutdown_connections 2024-09-11T05:07:29.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- 172.21.15.190:0/4120637343 wait complete. 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient: build_initial_monmap 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient: monmap: 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.460 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 librados: starting msgr at 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 librados: starting objecter 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 Processor -- start 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- start start 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 librados: setting wanted keys 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 librados: calling monclient init 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient: init 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f761006d3a0 0x7f761012ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient(hunting): picked mon.noname-b con 0x7f761006d3a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76100748d0 0x7f761012f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient(hunting): picked mon.noname-a con 0x7f76100748d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100a3c30 0x7f7610133cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient(hunting): picked mon.noname-c con 0x7f76100a3c30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7610143380 con 0x7f76100748d0 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7610143200 con 0x7f761006d3a0 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7610143500 con 0x7f76100a3c30 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.449+0000 7f7615ffc640 10 monclient(hunting): authenticate will time out at 930.247985s 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f761006d3a0 0x7f761012ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f761006d3a0 0x7f761012ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:34866/0 (socket says 172.21.15.190:34866) 2024-09-11T05:07:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 -- 172.21.15.190:0/2326273205 learned_addr learned my addr 172.21.15.190:0/2326273205 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient(hunting): get_auth_request con 0x7f761006d3a0 auth_method 0 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760ffff640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100a3c30 0x7f7610133cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760effd640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76100748d0 0x7f761012f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760ffff640 10 monclient(hunting): get_auth_request con 0x7f76100a3c30 auth_method 0 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:29.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 -- 172.21.15.190:0/2326273205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100a3c30 msgr2=0x7f7610133cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100a3c30 0x7f7610133cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 -- 172.21.15.190:0/2326273205 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76100748d0 msgr2=0x7f761012f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76100748d0 0x7f761012f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 monclient: found mon.noname-b 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 -- 172.21.15.190:0/2326273205 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761013ffb0 con 0x7f761006d3a0 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 -- 172.21.15.190:0/2326273205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f761006d3a0 msgr2=0x7f761012ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760f7fe640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f761006d3a0 0x7f761012ebd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f760801d090 tx=0x7f7608019590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 client.?.objecter ms_handle_connect 0x7f761006d3a0 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 1 -- 172.21.15.190:0/2326273205 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76080c0070 con 0x7f761006d3a0 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: dump: 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: _finish_auth 0 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:29.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 5 monclient: authenticate success, global_id 4123 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 1 -- 172.21.15.190:0/2326273205 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76080a6d50 con 0x7f761006d3a0 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 10 monclient: _renew_subs 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761013b070 con 0x7f761006d3a0 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 1 -- 172.21.15.190:0/2326273205 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76080ac050 con 0x7f761006d3a0 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 monclient: dump: 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:29.466 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f760dffb640 4 set_mon_vals no callback set 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 10 monclient: _renew_subs 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f761013ad90 con 0x7f761006d3a0 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 1 librados: init done 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 1 -- 172.21.15.190:0/2326273205 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f76080bb6b0 con 0x7f761006d3a0 2024-09-11T05:07:29.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 1 -- 172.21.15.190:0/2326273205 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f76080b1bb0 con 0x7f761006d3a0 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 10 client.4123.objecter ms_dispatch 0x7f7610000ba0 osd_map(5..5 src has 1..5) 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 3 client.4123.objecter handle_osd_map got epochs [5,5] > 0 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 3 client.4123.objecter handle_osd_map decoding full epoch 5 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f75f7fff640 20 client.4123.objecter dump_active .. 0 homeless 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.453+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76100a40c0 con 0x7f761006d3a0 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.457+0000 7f75f7fff640 1 -- 172.21.15.190:0/2326273205 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f7608016500 con 0x7f761006d3a0 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.457+0000 7f75f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:29.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.457+0000 7f75f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:29.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.529+0000 7f7615ffc640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e1a7ea3f-62f5-4d1f-b671-be183143eb20", "id": 3}] 2024-09-11T05:07:29.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.529+0000 7f7615ffc640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e1a7ea3f-62f5-4d1f-b671-be183143eb20", "id": 3}] 2024-09-11T05:07:29.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.529+0000 7f7615ffc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:29.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:29.529+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e1a7ea3f-62f5-4d1f-b671-be183143eb20", "id": 3} v 0) -- 0x7f7610068df0 con 0x7f761006d3a0 2024-09-11T05:07:30.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.269+0000 7f75f7fff640 1 -- 172.21.15.190:0/2326273205 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1a7ea3f-62f5-4d1f-b671-be183143eb20", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f76080bbbb0 con 0x7f761006d3a0 2024-09-11T05:07:30.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.269+0000 7f75f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e1a7ea3f-62f5-4d1f-b671-be183143eb20", "id": 3}] 2024-09-11T05:07:30.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.269+0000 7f75f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:30.274 INFO:teuthology.orchestra.run.smithi190.stdout:3 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 10 librados: watch_flush enter 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 10 librados: watch_flush exit 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 20 client.4123.objecter shutdown clearing up homeless session... 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 10 client.4123.objecter successfully canceled tick 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 10 monclient: shutdown 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f761006d3a0 msgr2=0x7f761012ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f761006d3a0 0x7f761012ebd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f760801d090 tx=0x7f7608019590 comp rx=0 tx=0).stop 2024-09-11T05:07:30.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 shutdown_connections 2024-09-11T05:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f76100a3c30 0x7f7610133cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f76100748d0 0x7f761012f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 --2- 172.21.15.190:0/2326273205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f761006d3a0 0x7f761012ebd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 >> 172.21.15.190:0/2326273205 conn(0x7f76100a55d0 msgr2=0x7f7610137480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 shutdown_connections 2024-09-11T05:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 -- 172.21.15.190:0/2326273205 wait complete. 2024-09-11T05:07:30.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.273+0000 7f7615ffc640 1 librados: shutdown 2024-09-11T05:07:30.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new a1aca44b-3dce-4f17-bc06-69bdca9cba04 4 2024-09-11T05:07:30.427 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient: get_monmap_and_config 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient: build_initial_monmap 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient: monmap: 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:07:30.429658+0000 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:07:30.429658+0000 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:30.428 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:30.429 INFO:teuthology.orchestra.run.smithi190.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:30.429 INFO:teuthology.orchestra.run.smithi190.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:30.429 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.429 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 1 Processor -- start 2024-09-11T05:07:30.429 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 1 -- start start 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient: init 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7faf4c0a1740 0x7faf4c0a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient(hunting): picked mon.noname-a con 0x7faf4c0a1740 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf4c09fa30 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf4c0a2060 0x7faf4c13d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient(hunting): picked mon.noname-c con 0x7faf4c0a2060 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf4c09e760 con 0x7faf4c09fa30 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7faf4c09f090 con 0x7faf4c0a1740 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7faf4c09ea40 con 0x7faf4c0a2060 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf53edf640 10 monclient(hunting): authenticate will time out at 931.223125s 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf4c0a2060 0x7faf4c13d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faf4c0a2060 0x7faf4c13d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:50970/0 (socket says 172.21.15.190:50970) 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 1 -- 172.21.15.190:0/1449058249 learned_addr learned my addr 172.21.15.190:0/1449058249 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 10 monclient(hunting): get_auth_request con 0x7faf4c0a2060 auth_method 0 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:30.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.425+0000 7faf5247c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:30.434 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-09-11T05:07:30.434 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 1 -- 172.21.15.190:0/1449058249 >> v2:172.21.15.138:3300/0 conn(0x7faf4c0a1740 msgr2=0x7faf4c0a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:30.434 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 1 --2- 172.21.15.190:0/1449058249 >> v2:172.21.15.138:3300/0 conn(0x7faf4c0a1740 0x7faf4c0a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.434 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 1 -- 172.21.15.190:0/1449058249 >> v1:172.21.15.138:6789/0 conn(0x7faf4c09fa30 legacy=0x7faf4c13cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:30.434 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 10 monclient: _finish_hunting 0 2024-09-11T05:07:30.434 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 1 monclient: found mon.noname-c 2024-09-11T05:07:30.434 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:30.435 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.435 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.429+0000 7faf5247c640 1 -- 172.21.15.190:0/1449058249 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4c09ed20 con 0x7faf4c0a2060 2024-09-11T05:07:30.436 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf5247c640 1 -- 172.21.15.190:0/1449058249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c0a2060 msgr2=0x7faf4c13d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:30.436 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf5247c640 1 --2- 172.21.15.190:0/1449058249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c0a2060 0x7faf4c13d500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faf4c074100 tx=0x7faf4401c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faf91050ff5bfdfc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:30.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 1 -- 172.21.15.190:0/1449058249 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf440a5770 con 0x7faf4c0a2060 2024-09-11T05:07:30.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:30.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: dump: 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: _finish_auth 0 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: _check_auth_tickets 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf53edf640 5 monclient: authenticate success, global_id 4126 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 1 -- 172.21.15.190:0/1449058249 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf440a5910 con 0x7faf4c0a2060 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf53edf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf53edf640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf53edf640 4 set_mon_vals no callback set 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 1 -- 172.21.15.190:0/1449058249 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf440a5cc0 con 0x7faf4c0a2060 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:30.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.433+0000 7faf50c79640 10 monclient: dump: 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:30.440 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 10 monclient: shutdown 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 -- 172.21.15.190:0/1449058249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c0a2060 msgr2=0x7faf4c13d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 --2- 172.21.15.190:0/1449058249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c0a2060 0x7faf4c13d500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faf4c074100 tx=0x7faf4401c8d0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 -- 172.21.15.190:0/1449058249 shutdown_connections 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 --2- 172.21.15.190:0/1449058249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c0a2060 0x7faf4c13d500 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 --2- 172.21.15.190:0/1449058249 >> v2:172.21.15.138:3300/0 conn(0x7faf4c0a1740 0x7faf4c0a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 -- 172.21.15.190:0/1449058249 >> 172.21.15.190:0/1449058249 conn(0x7faf4c0a74f0 msgr2=0x7faf4c0a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 -- 172.21.15.190:0/1449058249 shutdown_connections 2024-09-11T05:07:30.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 -- 172.21.15.190:0/1449058249 wait complete. 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 10 monclient: build_initial_monmap 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 10 monclient: monmap: 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 librados: starting msgr at 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 librados: starting objecter 2024-09-11T05:07:30.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 Processor -- start 2024-09-11T05:07:30.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 -- start start 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 librados: setting wanted keys 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 1 librados: calling monclient init 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.437+0000 7faf53edf640 10 monclient: init 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c09fa30 0x7faf4c132cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient(hunting): picked mon.noname-b con 0x7faf4c09fa30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faf4c0a1740 0x7faf4c141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient(hunting): picked mon.noname-a con 0x7faf4c0a1740 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faf4c0a2060 0x7faf4c1415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient(hunting): picked mon.noname-c con 0x7faf4c0a2060 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7faf4c13dbc0 con 0x7faf4c0a1740 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faf4c13da40 con 0x7faf4c09fa30 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7faf4c1458d0 con 0x7faf4c0a2060 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf53edf640 10 monclient(hunting): authenticate will time out at 931.236253s 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c09fa30 0x7faf4c132cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c09fa30 0x7faf4c132cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:50978/0 (socket says 172.21.15.190:50978) 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 -- 172.21.15.190:0/3872679972 learned_addr learned my addr 172.21.15.190:0/3872679972 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient(hunting): get_auth_request con 0x7faf4c09fa30 auth_method 0 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:30.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf5247c640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faf4c0a2060 0x7faf4c1415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf5247c640 10 monclient(hunting): get_auth_request con 0x7faf4c0a2060 auth_method 0 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf5247c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf5247c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf5247c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 -- 172.21.15.190:0/3872679972 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faf4c0a2060 msgr2=0x7faf4c1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faf4c0a2060 0x7faf4c1415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 -- 172.21.15.190:0/3872679972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faf4c0a1740 msgr2=0x7faf4c141090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faf4c0a1740 0x7faf4c141090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient: _finish_hunting 0 2024-09-11T05:07:30.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 monclient: found mon.noname-b 2024-09-11T05:07:30.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:30.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 -- 172.21.15.190:0/3872679972 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4c0a5060 con 0x7faf4c09fa30 2024-09-11T05:07:30.447 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf5247c640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faf4c0a2060 0x7faf4c1415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:07:30.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 -- 172.21.15.190:0/3872679972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c09fa30 msgr2=0x7faf4c132cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:30.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf51c7b640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c09fa30 0x7faf4c132cc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf4800b580 tx=0x7faf4800ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:30.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf427fc640 10 client.?.objecter ms_handle_connect 0x7faf4c09fa30 2024-09-11T05:07:30.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf427fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.441+0000 7faf427fc640 1 -- 172.21.15.190:0/3872679972 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48014070 con 0x7faf4c09fa30 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: dump: 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:30.449 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 1 -- 172.21.15.190:0/3872679972 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf48010aa0 con 0x7faf4c09fa30 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 1 -- 172.21.15.190:0/3872679972 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4800f3c0 con 0x7faf4c09fa30 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 monclient: dump: 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.450 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf437fe640 4 set_mon_vals no callback set 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 5 monclient: authenticate success, global_id 4129 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 10 monclient: _renew_subs 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf4c09e0a0 con 0x7faf4c09fa30 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 10 monclient: _renew_subs 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf4c144040 con 0x7faf4c09fa30 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 1 librados: init done 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 1 -- 172.21.15.190:0/3872679972 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7faf4800f560 con 0x7faf4c09fa30 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 1 -- 172.21.15.190:0/3872679972 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7faf48017360 con 0x7faf4c09fa30 2024-09-11T05:07:30.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 10 client.4129.objecter ms_dispatch 0x7faf4c000ba0 osd_map(6..6 src has 1..6) 2024-09-11T05:07:30.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 3 client.4129.objecter handle_osd_map got epochs [6,6] > 0 2024-09-11T05:07:30.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 3 client.4129.objecter handle_osd_map decoding full epoch 6 2024-09-11T05:07:30.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf427fc640 20 client.4129.objecter dump_active .. 0 homeless 2024-09-11T05:07:30.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:30.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:30.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.445+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf4c145ce0 con 0x7faf4c09fa30 2024-09-11T05:07:30.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.449+0000 7faf427fc640 1 -- 172.21.15.190:0/3872679972 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7faf48019450 con 0x7faf4c09fa30 2024-09-11T05:07:30.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.449+0000 7faf427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:30.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.449+0000 7faf427fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:30.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.521+0000 7faf53edf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a1aca44b-3dce-4f17-bc06-69bdca9cba04", "id": 4}] 2024-09-11T05:07:30.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.521+0000 7faf53edf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a1aca44b-3dce-4f17-bc06-69bdca9cba04", "id": 4}] 2024-09-11T05:07:30.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.521+0000 7faf53edf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.521+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a1aca44b-3dce-4f17-bc06-69bdca9cba04", "id": 4} v 0) -- 0x7faf4c0a2950 con 0x7faf4c09fa30 2024-09-11T05:07:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf427fc640 1 -- 172.21.15.190:0/3872679972 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a1aca44b-3dce-4f17-bc06-69bdca9cba04", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7faf480199f0 con 0x7faf4c09fa30 2024-09-11T05:07:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a1aca44b-3dce-4f17-bc06-69bdca9cba04", "id": 4}] 2024-09-11T05:07:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf427fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:30.535 INFO:teuthology.orchestra.run.smithi190.stdout:4 2024-09-11T05:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 10 librados: watch_flush enter 2024-09-11T05:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 10 librados: watch_flush exit 2024-09-11T05:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 20 client.4129.objecter shutdown clearing up homeless session... 2024-09-11T05:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 10 client.4129.objecter successfully canceled tick 2024-09-11T05:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 10 monclient: shutdown 2024-09-11T05:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c09fa30 msgr2=0x7faf4c132cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:30.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c09fa30 0x7faf4c132cc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf4800b580 tx=0x7faf4800ba40 comp rx=0 tx=0).stop 2024-09-11T05:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 shutdown_connections 2024-09-11T05:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faf4c0a2060 0x7faf4c1415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faf4c0a1740 0x7faf4c141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 --2- 172.21.15.190:0/3872679972 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faf4c09fa30 0x7faf4c132cc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 >> 172.21.15.190:0/3872679972 conn(0x7faf4c0a74f0 msgr2=0x7faf4c0a5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 shutdown_connections 2024-09-11T05:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 -- 172.21.15.190:0/3872679972 wait complete. 2024-09-11T05:07:30.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.533+0000 7faf53edf640 1 librados: shutdown 2024-09-11T05:07:30.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 87ad0149-d931-4ac6-8787-94f9a9609d86 5 2024-09-11T05:07:30.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient: get_monmap_and_config 2024-09-11T05:07:30.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient: build_initial_monmap 2024-09-11T05:07:30.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient: monmap: 2024-09-11T05:07:30.664 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:30.664 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.664 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:07:30.667117+0000 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:07:30.667117+0000 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:30.665 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 1 Processor -- start 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 1 -- start start 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient: init 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd92009da20 0x7fd9200a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient(hunting): picked mon.noname-b con 0x7fd92009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd9200a1f40 0x7fd9200a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient(hunting): picked mon.noname-a con 0x7fd9200a1f40 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd920136170 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd920074a80 con 0x7fd920136170 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fd9200a2d90 con 0x7fd9200a1f40 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd9200a3070 con 0x7fd92009da20 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd926a6c640 10 monclient(hunting): authenticate will time out at 931.459321s 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd925009640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd920136170 0x7fd920136600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:38084/0 (socket says 172.21.15.190:38084) 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd925009640 1 -- 172.21.15.190:0/3198156365 learned_addr learned my addr 172.21.15.190:0/3198156365 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd924808640 1 --2- 172.21.15.190:0/3198156365 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd92009da20 0x7fd9200a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd924808640 10 monclient(hunting): get_auth_request con 0x7fd92009da20 auth_method 0 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd924808640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd924808640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd924808640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd91ffff640 1 --2- 172.21.15.190:0/3198156365 >> v2:172.21.15.138:3300/0 conn(0x7fd9200a1f40 0x7fd9200a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd91ffff640 10 monclient(hunting): get_auth_request con 0x7fd9200a1f40 auth_method 0 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd91ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:30.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd91ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd924808640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd924808640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.661+0000 7fd924808640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd91ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd91ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd91ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 1 -- 172.21.15.190:0/3198156365 >> v2:172.21.15.138:3300/0 conn(0x7fd9200a1f40 msgr2=0x7fd9200a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 1 --2- 172.21.15.190:0/3198156365 >> v2:172.21.15.138:3300/0 conn(0x7fd9200a1f40 0x7fd9200a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 1 -- 172.21.15.190:0/3198156365 >> v1:172.21.15.190:6789/0 conn(0x7fd920136170 legacy=0x7fd920136600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 10 monclient: _finish_hunting 0 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 1 monclient: found mon.noname-b 2024-09-11T05:07:30.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:30.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:30.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.665+0000 7fd924808640 1 -- 172.21.15.190:0/3198156365 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9200a3350 con 0x7fd92009da20 2024-09-11T05:07:30.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd924808640 1 -- 172.21.15.190:0/3198156365 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd92009da20 msgr2=0x7fd9200a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:30.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd924808640 1 --2- 172.21.15.190:0/3198156365 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd92009da20 0x7fd9200a1a00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd9140097c0 tx=0x7fd91401c210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52e3c1da820bffe server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 1 -- 172.21.15.190:0/3198156365 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9140b0070 con 0x7fd92009da20 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: dump: 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 1 -- 172.21.15.190:0/3198156365 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9140ab600 con 0x7fd92009da20 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 1 -- 172.21.15.190:0/3198156365 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9140ab9d0 con 0x7fd92009da20 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:30.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd91f7fe640 10 monclient: dump: 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 5 monclient: authenticate success, global_id 4115 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:30.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 4 set_mon_vals no callback set 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 10 monclient: shutdown 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 1 -- 172.21.15.190:0/3198156365 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd92009da20 msgr2=0x7fd9200a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 1 --2- 172.21.15.190:0/3198156365 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd92009da20 0x7fd9200a1a00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd9140097c0 tx=0x7fd91401c210 comp rx=0 tx=0).stop 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 1 -- 172.21.15.190:0/3198156365 shutdown_connections 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 1 --2- 172.21.15.190:0/3198156365 >> v2:172.21.15.138:3300/0 conn(0x7fd9200a1f40 0x7fd9200a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 1 --2- 172.21.15.190:0/3198156365 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd92009da20 0x7fd9200a1a00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 1 -- 172.21.15.190:0/3198156365 >> 172.21.15.190:0/3198156365 conn(0x7fd9200a3b00 msgr2=0x7fd9200a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 1 -- 172.21.15.190:0/3198156365 shutdown_connections 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 1 -- 172.21.15.190:0/3198156365 wait complete. 2024-09-11T05:07:30.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 10 monclient: build_initial_monmap 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.669+0000 7fd926a6c640 10 monclient: monmap: 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 librados: starting msgr at 2024-09-11T05:07:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 librados: starting objecter 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 Processor -- start 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 -- start start 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 librados: setting wanted keys 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 librados: calling monclient init 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient: init 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd92009da20 0x7fd92012fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient(hunting): picked mon.noname-a con 0x7fd92009da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 0x7fd9200738d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient(hunting): picked mon.noname-b con 0x7fd9200a1f40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd920136170 0x7fd920073e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient(hunting): picked mon.noname-c con 0x7fd920136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd9201410c0 con 0x7fd92009da20 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd920140f40 con 0x7fd9200a1f40 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd920141240 con 0x7fd920136170 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient(hunting): authenticate will time out at 931.468936s 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 0x7fd9200738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 0x7fd9200738d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:50990/0 (socket says 172.21.15.190:50990) 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 -- 172.21.15.190:0/680073789 learned_addr learned my addr 172.21.15.190:0/680073789 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient(hunting): get_auth_request con 0x7fd9200a1f40 auth_method 0 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:30.682 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd925009640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd920136170 0x7fd920073e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd924808640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd92009da20 0x7fd92012fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd925009640 10 monclient(hunting): get_auth_request con 0x7fd920136170 auth_method 0 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd925009640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd925009640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd925009640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd924808640 10 monclient(hunting): get_auth_request con 0x7fd92009da20 auth_method 0 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd924808640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd924808640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd924808640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-11T05:07:30.683 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 -- 172.21.15.190:0/680073789 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd920136170 msgr2=0x7fd920073e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd920136170 0x7fd920073e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 -- 172.21.15.190:0/680073789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd92009da20 msgr2=0x7fd92012fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd92009da20 0x7fd92012fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 monclient: found mon.noname-b 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 -- 172.21.15.190:0/680073789 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9200712a0 con 0x7fd9200a1f40 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 -- 172.21.15.190:0/680073789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 msgr2=0x7fd9200738d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91ffff640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 0x7fd9200738d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd91000e930 tx=0x7fd91000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 client.?.objecter ms_handle_connect 0x7fd9200a1f40 2024-09-11T05:07:30.684 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 1 -- 172.21.15.190:0/680073789 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910017070 con 0x7fd9200a1f40 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: dump: 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: _finish_auth 0 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 1 -- 172.21.15.190:0/680073789 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91000cc10 con 0x7fd9200a1f40 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 1 -- 172.21.15.190:0/680073789 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910013470 con 0x7fd9200a1f40 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91d7fa640 10 monclient: dump: 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:30.686 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 5 monclient: authenticate success, global_id 4132 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient: _renew_subs 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd91effd640 4 set_mon_vals no callback set 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.673+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd920135f00 con 0x7fd9200a1f40 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 10 monclient: _renew_subs 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd920070ba0 con 0x7fd9200a1f40 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 1 librados: init done 2024-09-11T05:07:30.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd91d7fa640 1 -- 172.21.15.190:0/680073789 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fd910024060 con 0x7fd9200a1f40 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd91d7fa640 1 -- 172.21.15.190:0/680073789 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fd910013d40 con 0x7fd9200a1f40 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd91d7fa640 10 client.4132.objecter ms_dispatch 0x7fd920000ba0 osd_map(7..7 src has 1..7) 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd91d7fa640 3 client.4132.objecter handle_osd_map got epochs [7,7] > 0 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd91d7fa640 3 client.4132.objecter handle_osd_map decoding full epoch 7 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd91d7fa640 20 client.4132.objecter dump_active .. 0 homeless 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9200a1a00 con 0x7fd9200a1f40 2024-09-11T05:07:30.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:30.677+0000 7fd91ffff640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 0x7fd9200738d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd91000e930 tx=0x7fd91000edf0 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7fd910013610 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-11T05:07:30.721 INFO:tasks.ceph.mgr.x.smithi190.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T05:07:30.721 INFO:tasks.ceph.mgr.x.smithi190.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T05:07:30.721 INFO:tasks.ceph.mgr.x.smithi190.stderr: from numpy import show_config as show_numpy_config 2024-09-11T05:07:31.509 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.505+0000 7fd91d7fa640 1 -- 172.21.15.190:0/680073789 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fd910013610 con 0x7fd9200a1f40 2024-09-11T05:07:31.509 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.505+0000 7fd91d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:31.509 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.505+0000 7fd91d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:31.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.581+0000 7fd926a6c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "87ad0149-d931-4ac6-8787-94f9a9609d86", "id": 5}] 2024-09-11T05:07:31.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.581+0000 7fd926a6c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "87ad0149-d931-4ac6-8787-94f9a9609d86", "id": 5}] 2024-09-11T05:07:31.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.581+0000 7fd926a6c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:07:31.582 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.581+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd new", "uuid": "87ad0149-d931-4ac6-8787-94f9a9609d86", "id": 5} v 0) -- 0x7fd92006cd10 con 0x7fd9200a1f40 2024-09-11T05:07:31.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd91d7fa640 1 -- 172.21.15.190:0/680073789 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87ad0149-d931-4ac6-8787-94f9a9609d86", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fd9100059a0 con 0x7fd9200a1f40 2024-09-11T05:07:31.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd91d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "87ad0149-d931-4ac6-8787-94f9a9609d86", "id": 5}] 2024-09-11T05:07:31.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd91d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:31.588 INFO:teuthology.orchestra.run.smithi190.stdout:5 2024-09-11T05:07:31.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 10 librados: watch_flush enter 2024-09-11T05:07:31.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 10 librados: watch_flush exit 2024-09-11T05:07:31.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 20 client.4132.objecter shutdown clearing up homeless session... 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 10 client.4132.objecter successfully canceled tick 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 10 monclient: shutdown 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 msgr2=0x7fd9200738d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 0x7fd9200738d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd91000e930 tx=0x7fd91000edf0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 shutdown_connections 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd920136170 0x7fd920073e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9200a1f40 0x7fd9200738d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 --2- 172.21.15.190:0/680073789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd92009da20 0x7fd92012fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 >> 172.21.15.190:0/680073789 conn(0x7fd9200a3b00 msgr2=0x7fd92012c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 shutdown_connections 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 -- 172.21.15.190:0/680073789 wait complete. 2024-09-11T05:07:31.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T05:07:31.585+0000 7fd926a6c640 1 librados: shutdown 2024-09-11T05:07:31.605 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T05:07:31.605 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T05:07:31.608 INFO:tasks.ceph.osd.0:Started 2024-09-11T05:07:31.608 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T05:07:31.608 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T05:07:31.611 INFO:tasks.ceph.osd.1:Started 2024-09-11T05:07:31.611 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T05:07:31.611 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-11T05:07:31.613 INFO:tasks.ceph.osd.2:Started 2024-09-11T05:07:31.614 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T05:07:31.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T05:07:31.616 INFO:tasks.ceph.osd.3:Started 2024-09-11T05:07:31.616 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-11T05:07:31.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-11T05:07:31.617 INFO:tasks.ceph.osd.4:Started 2024-09-11T05:07:31.618 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-11T05:07:31.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-11T05:07:31.619 INFO:tasks.ceph.osd.5:Started 2024-09-11T05:07:31.620 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:07:31.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.731+0000 7fb849db6640 10 monclient: get_monmap_and_config 2024-09-11T05:07:31.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.731+0000 7fb849db6640 10 monclient: build_initial_monmap 2024-09-11T05:07:31.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.731+0000 7fb849db6640 10 monclient: monmap: 2024-09-11T05:07:31.735 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:31.735 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:31.736555+0000 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:31.736555+0000 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.731+0000 7fb849db6640 1 Processor -- start 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.731+0000 7fb849db6640 1 -- start start 2024-09-11T05:07:31.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.731+0000 7fb849db6640 10 monclient: init 2024-09-11T05:07:31.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb8440a5b40 0x7fb8440a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient(hunting): picked mon.noname-b con 0x7fb8440a5b40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb8440a7130 0x7fb8440a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient(hunting): picked mon.noname-c con 0x7fb8440a7130 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb84409d840 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb84409d490 con 0x7fb84409d840 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb84409e110 con 0x7fb8440a7130 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb8440a69a0 con 0x7fb8440a5b40 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:31.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb849db6640 10 monclient(hunting): authenticate will time out at 935.172400s 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb8440a5b40 0x7fb8440a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb843fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fb84409d840 0x7fb844138900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:56114/0 (socket says 172.21.15.138:56114) 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb8440a5b40 0x7fb8440a5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:39154/0 (socket says 172.21.15.138:39154) 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 1 -- 172.21.15.138:0/2021949459 learned_addr learned my addr 172.21.15.138:0/2021949459 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient(hunting): get_auth_request con 0x7fb8440a5b40 auth_method 0 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb83affd640 1 --2- 172.21.15.138:0/2021949459 >> v2:172.21.15.190:3300/0 conn(0x7fb8440a7130 0x7fb8440a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:31.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb83affd640 10 monclient(hunting): get_auth_request con 0x7fb8440a7130 auth_method 0 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb83affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb83affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb83affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 1 -- 172.21.15.138:0/2021949459 >> v2:172.21.15.190:3300/0 conn(0x7fb8440a7130 msgr2=0x7fb8440a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 1 --2- 172.21.15.138:0/2021949459 >> v2:172.21.15.190:3300/0 conn(0x7fb8440a7130 0x7fb8440a6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 1 -- 172.21.15.138:0/2021949459 >> v1:172.21.15.138:6789/0 conn(0x7fb84409d840 legacy=0x7fb844138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 1 monclient: found mon.noname-b 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:31.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.735+0000 7fb8437fe640 1 -- 172.21.15.138:0/2021949459 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb844070580 con 0x7fb8440a5b40 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 1 -- 172.21.15.138:0/2021949459 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb8440a5b40 msgr2=0x7fb8440a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 1 --2- 172.21.15.138:0/2021949459 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb8440a5b40 0x7fb8440a5f20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb834009870 tx=0x7fb83401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b54b40bee9e53b5d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 1 -- 172.21.15.138:0/2021949459 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8340b0070 con 0x7fb8440a5b40 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: dump: 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:31.745 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 1 -- 172.21.15.138:0/2021949459 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8340ab600 con 0x7fb8440a5b40 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 5 monclient: authenticate success, global_id 4136 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:31.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 1 -- 172.21.15.138:0/2021949459 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8340ab9d0 con 0x7fb8440a5b40 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 4 set_mon_vals no callback set 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb842ffd640 10 monclient: dump: 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:31.747 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient: shutdown 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- 172.21.15.138:0/2021949459 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb8440a5b40 msgr2=0x7fb8440a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 --2- 172.21.15.138:0/2021949459 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb8440a5b40 0x7fb8440a5f20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb834009870 tx=0x7fb83401c420 comp rx=0 tx=0).stop 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- 172.21.15.138:0/2021949459 shutdown_connections 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 --2- 172.21.15.138:0/2021949459 >> v2:172.21.15.190:3300/0 conn(0x7fb8440a7130 0x7fb8440a6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 --2- 172.21.15.138:0/2021949459 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb8440a5b40 0x7fb8440a5f20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- 172.21.15.138:0/2021949459 >> 172.21.15.138:0/2021949459 conn(0x7fb8440aba10 msgr2=0x7fb8440a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- 172.21.15.138:0/2021949459 shutdown_connections 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- 172.21.15.138:0/2021949459 wait complete. 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient: build_initial_monmap 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient: monmap: 2024-09-11T05:07:31.748 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 librados: starting msgr at 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 librados: starting objecter 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 Processor -- start 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- start start 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 librados: setting wanted keys 2024-09-11T05:07:31.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 librados: calling monclient init 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient: init 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb84409d840 0x7fb84413e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient(hunting): picked mon.noname-c con 0x7fb84409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8440a5b40 0x7fb84413e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient(hunting): picked mon.noname-a con 0x7fb8440a5b40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8440a7130 0x7fb84413eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient(hunting): picked mon.noname-b con 0x7fb8440a7130 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb844142f00 con 0x7fb8440a5b40 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb844142d80 con 0x7fb8440a7130 2024-09-11T05:07:31.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb844143080 con 0x7fb84409d840 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb849db6640 10 monclient(hunting): authenticate will time out at 935.179287s 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb84409d840 0x7fb84413e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb84409d840 0x7fb84413e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:39164/0 (socket says 172.21.15.138:39164) 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 1 -- 172.21.15.138:0/253638402 learned_addr learned my addr 172.21.15.138:0/253638402 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8440a5b40 0x7fb84413e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 10 monclient(hunting): get_auth_request con 0x7fb84409d840 auth_method 0 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient(hunting): get_auth_request con 0x7fb8440a5b40 auth_method 0 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:31.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb843fff640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8440a7130 0x7fb84413eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient(hunting): handle_auth_done global_id 4134 payload 824 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 1 -- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb84409d840 msgr2=0x7fb84413e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb84409d840 0x7fb84413e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 1 -- 172.21.15.138:0/253638402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8440a7130 msgr2=0x7fb84413eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8440a7130 0x7fb84413eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:31.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 1 monclient: found mon.noname-a 2024-09-11T05:07:31.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:31.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:31.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb83affd640 1 -- 172.21.15.138:0/253638402 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84413fb00 con 0x7fb8440a5b40 2024-09-11T05:07:31.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb843fff640 10 monclient: get_auth_request con 0x7fb8440a7130 auth_method 0 2024-09-11T05:07:31.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb8437fe640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb84409d840 0x7fb84413e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:31.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.739+0000 7fb843fff640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8440a7130 0x7fb84413eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb83affd640 1 -- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8440a5b40 msgr2=0x7fb84413e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb83affd640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8440a5b40 0x7fb84413e580 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00d5f0 tx=0x7fb82c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 client.?.objecter ms_handle_connect 0x7fb8440a5b40 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 1 -- 172.21.15.138:0/253638402 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c012800 con 0x7fb8440a5b40 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: dump: 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:31.757 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: _finish_auth 0 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 5 monclient: authenticate success, global_id 4134 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 10 monclient: _renew_subs 2024-09-11T05:07:31.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 1 -- 172.21.15.138:0/253638402 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82c019070 con 0x7fb8440a5b40 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb844121c50 con 0x7fb8440a5b40 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 1 -- 172.21.15.138:0/253638402 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c012bc0 con 0x7fb8440a5b40 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 monclient: dump: 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:31.759 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 10 monclient: _renew_subs 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8441221f0 con 0x7fb8440a5b40 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 1 librados: init done 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb8427fc640 4 set_mon_vals no callback set 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 1 -- 172.21.15.138:0/253638402 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fb82c013070 con 0x7fb8440a5b40 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 1 -- 172.21.15.138:0/253638402 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fb82c012d60 con 0x7fb8440a5b40 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 10 client.4134.objecter ms_dispatch 0x7fb844000ba0 osd_map(8..8 src has 1..8) 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 3 client.4134.objecter handle_osd_map got epochs [8,8] > 0 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 3 client.4134.objecter handle_osd_map decoding full epoch 8 2024-09-11T05:07:31.760 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb840ff9640 20 client.4134.objecter dump_active .. 0 homeless 2024-09-11T05:07:31.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:31.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:31.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:31.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.743+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8440aaaa0 con 0x7fb8440a5b40 2024-09-11T05:07:31.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.747+0000 7fb840ff9640 1 -- 172.21.15.138:0/253638402 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fb844121c50 con 0x7fb8440a5b40 2024-09-11T05:07:31.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.747+0000 7fb840ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:31.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.747+0000 7fb840ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:31.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:31.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:31.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:31.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb844138900 con 0x7fb8440a5b40 2024-09-11T05:07:31.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb840ff9640 1 -- 172.21.15.138:0/253638402 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7fb82c01e4c0 con 0x7fb8440a5b40 2024-09-11T05:07:31.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb840ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:31.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb840ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:31.810 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:31.810 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":8,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","created":"2024-09-11T05:07:27.386635+0000","modified":"2024-09-11T05:07:31.585387+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:07:31.585387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79670fd2-b5aa-408e-8aa4-18a8fb848972","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e96fe3a6-3058-4c40-aa87-26a89572cdfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a87dbce5-b78c-4070-90cc-d323fc444f4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1a7ea3f-62f5-4d1f-b671-be183143eb20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a1aca44b-3dce-4f17-bc06-69bdca9cba04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"87ad0149-d931-4ac6-8787-94f9a9609d86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 10 librados: watch_flush enter 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 10 librados: watch_flush exit 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 20 client.4134.objecter shutdown clearing up homeless session... 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 10 client.4134.objecter successfully canceled tick 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 10 monclient: shutdown 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8440a5b40 msgr2=0x7fb84413e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8440a5b40 0x7fb84413e580 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00d5f0 tx=0x7fb82c00dab0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 shutdown_connections 2024-09-11T05:07:31.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8440a7130 0x7fb84413eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb8440a5b40 0x7fb84413e580 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 --2- 172.21.15.138:0/253638402 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb84409d840 0x7fb84413e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:31.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 >> 172.21.15.138:0/253638402 conn(0x7fb8440aba10 msgr2=0x7fb8440a4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:31.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 shutdown_connections 2024-09-11T05:07:31.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 -- 172.21.15.138:0/253638402 wait complete. 2024-09-11T05:07:31.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:31.807+0000 7fb849db6640 1 librados: shutdown 2024-09-11T05:07:31.825 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T05:07:31.826 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T05:07:32.430 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient: get_monmap_and_config 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient: build_initial_monmap 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient: monmap: 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:07:32.551546+0000 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:07:32.551546+0000 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:32.550 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:32.551 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:32.551 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:32.551 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:32.551 INFO:teuthology.misc.health.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:32.551 INFO:teuthology.misc.health.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:32.551 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 1 Processor -- start 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 1 -- start start 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient: init 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f052409da50 0x7f05241330f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient(hunting): picked mon.noname-a con 0x7f052409da50 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f05240745a0 0x7f05240749f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:32.553 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient(hunting): picked mon.noname-c con 0x7f05240745a0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05241361b0 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05240a2ea0 con 0x7f05241361b0 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f05240a0910 con 0x7f052409da50 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f05240a0ae0 con 0x7f05240745a0 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f052a493640 10 monclient(hunting): authenticate will time out at 935.987625s 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f0523fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f052409da50 0x7f05241330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:32.554 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f0528a30640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f05241361b0 0x7f0524136660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:56116/0 (socket says 172.21.15.138:56116) 2024-09-11T05:07:32.555 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f0528a30640 1 -- 172.21.15.138:0/1795371492 learned_addr learned my addr 172.21.15.138:0/1795371492 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:32.555 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f0523fff640 10 monclient(hunting): get_auth_request con 0x7f052409da50 auth_method 0 2024-09-11T05:07:32.555 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f0523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:32.555 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f0523fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.547+0000 7f0523fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f05237fe640 1 --2- 172.21.15.138:0/1795371492 >> v2:172.21.15.190:3300/0 conn(0x7f05240745a0 0x7f05240749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f05237fe640 10 monclient(hunting): get_auth_request con 0x7f05240745a0 auth_method 0 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f05237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f05237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f05237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:32.556 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 10 monclient(hunting): handle_auth_done global_id 4137 payload 274 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 1 -- 172.21.15.138:0/1795371492 >> v2:172.21.15.190:3300/0 conn(0x7f05240745a0 msgr2=0x7f05240749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 1 --2- 172.21.15.138:0/1795371492 >> v2:172.21.15.190:3300/0 conn(0x7f05240745a0 0x7f05240749f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 1 -- 172.21.15.138:0/1795371492 >> v1:172.21.15.138:6789/0 conn(0x7f05241361b0 legacy=0x7f0524136660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 1 monclient: found mon.noname-a 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:32.557 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.551+0000 7f0523fff640 1 -- 172.21.15.138:0/1795371492 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05240a3180 con 0x7f052409da50 2024-09-11T05:07:32.563 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 1 -- 172.21.15.138:0/1795371492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f052409da50 msgr2=0x7f05241330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:32.563 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 1 --2- 172.21.15.138:0/1795371492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f052409da50 0x7f05241330f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0524137230 tx=0x7f051401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6593429837a6b85f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:32.563 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 1 -- 172.21.15.138:0/1795371492 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05140b0070 con 0x7f052409da50 2024-09-11T05:07:32.563 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:32.563 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: dump: 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:32.564 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 5 monclient: authenticate success, global_id 4137 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 1 -- 172.21.15.138:0/1795371492 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05140ab600 con 0x7f052409da50 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 1 -- 172.21.15.138:0/1795371492 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05140ab9d0 con 0x7f052409da50 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:32.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0522ffd640 10 monclient: dump: 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 4 set_mon_vals no callback set 2024-09-11T05:07:32.566 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient: shutdown 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- 172.21.15.138:0/1795371492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f052409da50 msgr2=0x7f05241330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 --2- 172.21.15.138:0/1795371492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f052409da50 0x7f05241330f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0524137230 tx=0x7f051401c340 comp rx=0 tx=0).stop 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- 172.21.15.138:0/1795371492 shutdown_connections 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 --2- 172.21.15.138:0/1795371492 >> v2:172.21.15.190:3300/0 conn(0x7f05240745a0 0x7f05240749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 --2- 172.21.15.138:0/1795371492 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f052409da50 0x7f05241330f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- 172.21.15.138:0/1795371492 >> 172.21.15.138:0/1795371492 conn(0x7f05240a38e0 msgr2=0x7f05240a3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- 172.21.15.138:0/1795371492 shutdown_connections 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- 172.21.15.138:0/1795371492 wait complete. 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient: build_initial_monmap 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient: monmap: 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:32.567 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 librados: starting msgr at 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 librados: starting objecter 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 Processor -- start 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- start start 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 librados: setting wanted keys 2024-09-11T05:07:32.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 librados: calling monclient init 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient: init 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05240745a0 0x7f052413d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient(hunting): picked mon.noname-a con 0x7f05240745a0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052409da50 0x7f052413d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient(hunting): picked mon.noname-b con 0x7f052409da50 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f05241361b0 0x7f0524138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient(hunting): picked mon.noname-c con 0x7f05241361b0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0524141100 con 0x7f05240745a0 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:32.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0524140f80 con 0x7f052409da50 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0524141280 con 0x7f05241361b0 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f052a493640 10 monclient(hunting): authenticate will time out at 935.995091s 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05240745a0 0x7f052413d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f05241361b0 0x7f0524138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05240745a0 0x7f052413d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:50858/0 (socket says 172.21.15.138:50858) 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 1 -- 172.21.15.138:0/298082737 learned_addr learned my addr 172.21.15.138:0/298082737 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 10 monclient(hunting): get_auth_request con 0x7f05240745a0 auth_method 0 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:32.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient(hunting): get_auth_request con 0x7f05241361b0 auth_method 0 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f05237fe640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052409da50 0x7f052413d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 1 -- 172.21.15.138:0/298082737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052409da50 msgr2=0x7f052413d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:32.571 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052409da50 0x7f052413d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 1 -- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05240745a0 msgr2=0x7f052413d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05240745a0 0x7f052413d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient: _finish_hunting 0 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 1 monclient: found mon.noname-c 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0528a30640 1 -- 172.21.15.138:0/298082737 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052413e2a0 con 0x7f05241361b0 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f05237fe640 10 monclient: get_auth_request con 0x7f052409da50 auth_method 0 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f0523fff640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05240745a0 0x7f052413d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.555+0000 7f05237fe640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052409da50 0x7f052413d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0528a30640 1 -- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f05241361b0 msgr2=0x7f0524138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0528a30640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f05241361b0 0x7f0524138200 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f051c00cb20 tx=0x7f051c00cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 client.?.objecter ms_handle_connect 0x7f05241361b0 2024-09-11T05:07:32.572 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 1 -- 172.21.15.138:0/298082737 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c017070 con 0x7f05241361b0 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: dump: 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:32.573 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:32.574 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:32.574 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:32.574 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:32.577 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:32.577 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:32.577 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: _finish_auth 0 2024-09-11T05:07:32.577 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 1 -- 172.21.15.138:0/298082737 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f051c00ade0 con 0x7f05241361b0 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 1 -- 172.21.15.138:0/298082737 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c012640 con 0x7f05241361b0 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 5 monclient: authenticate success, global_id 4142 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 10 monclient: _renew_subs 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f052413de40 con 0x7f05241361b0 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 monclient: dump: 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:32.578 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f05227fc640 4 set_mon_vals no callback set 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 10 monclient: _renew_subs 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:32.579 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f052413be90 con 0x7f05241361b0 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 1 librados: init done 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 1 -- 172.21.15.138:0/298082737 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f051c012820 con 0x7f05241361b0 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 1 -- 172.21.15.138:0/298082737 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f051c007660 con 0x7f05241361b0 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 10 client.4142.objecter ms_dispatch 0x7f0524000ba0 osd_map(8..8 src has 1..8) 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 3 client.4142.objecter handle_osd_map got epochs [8,8] > 0 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 3 client.4142.objecter handle_osd_map decoding full epoch 8 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f0520ff9640 20 client.4142.objecter dump_active .. 0 homeless 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:32.580 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.559+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05240749f0 con 0x7f05241361b0 2024-09-11T05:07:32.581 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.563+0000 7f0520ff9640 1 -- 172.21.15.138:0/298082737 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f051c021070 con 0x7f05241361b0 2024-09-11T05:07:32.581 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.563+0000 7f0520ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:32.581 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.563+0000 7f0520ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:32.624 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:32.625 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:32.625 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:32.625 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f052413a690 con 0x7f05241361b0 2024-09-11T05:07:32.625 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f0520ff9640 1 -- 172.21.15.138:0/298082737 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f051c012d20 con 0x7f05241361b0 2024-09-11T05:07:32.625 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f0520ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:32.626 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f0520ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:32.626 INFO:teuthology.misc.health.smithi138.stdout: 2024-09-11T05:07:32.626 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":8,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","created":"2024-09-11T05:07:27.386635+0000","modified":"2024-09-11T05:07:31.585387+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:07:31.585387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79670fd2-b5aa-408e-8aa4-18a8fb848972","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e96fe3a6-3058-4c40-aa87-26a89572cdfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a87dbce5-b78c-4070-90cc-d323fc444f4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1a7ea3f-62f5-4d1f-b671-be183143eb20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a1aca44b-3dce-4f17-bc06-69bdca9cba04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"87ad0149-d931-4ac6-8787-94f9a9609d86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:07:32.627 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 10 librados: watch_flush enter 2024-09-11T05:07:32.627 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 10 librados: watch_flush exit 2024-09-11T05:07:32.627 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 20 client.4142.objecter shutdown clearing up homeless session... 2024-09-11T05:07:32.627 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 10 client.4142.objecter successfully canceled tick 2024-09-11T05:07:32.627 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 10 monclient: shutdown 2024-09-11T05:07:32.627 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:32.627 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f05241361b0 msgr2=0x7f0524138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:32.627 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f05241361b0 0x7f0524138200 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f051c00cb20 tx=0x7f051c00cf20 comp rx=0 tx=0).stop 2024-09-11T05:07:32.628 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 shutdown_connections 2024-09-11T05:07:32.628 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f05241361b0 0x7f0524138200 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:32.628 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052409da50 0x7f052413d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:32.628 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 --2- 172.21.15.138:0/298082737 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f05240745a0 0x7f052413d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:32.628 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 >> 172.21.15.138:0/298082737 conn(0x7f05240a38e0 msgr2=0x7f052409ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:32.628 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 shutdown_connections 2024-09-11T05:07:32.628 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 -- 172.21.15.138:0/298082737 wait complete. 2024-09-11T05:07:32.628 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:32.623+0000 7f052a493640 1 librados: shutdown 2024-09-11T05:07:32.641 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-11T05:07:32.785 INFO:tasks.ceph.osd.0.smithi138.stderr:2024-09-11T05:07:32.783+0000 7f2eceaa57c0 -1 Falling back to public interface 2024-09-11T05:07:32.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2024-09-11T05:07:32.799+0000 7f72688687c0 -1 Falling back to public interface 2024-09-11T05:07:32.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2024-09-11T05:07:32.959+0000 7fe9ea20e7c0 -1 Falling back to public interface 2024-09-11T05:07:33.472 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T05:07:33.469+0000 7ff6212007c0 -1 Falling back to public interface 2024-09-11T05:07:33.535 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-09-11T05:07:33.533+0000 7f6f7decf7c0 -1 Falling back to public interface 2024-09-11T05:07:33.547 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T05:07:33.545+0000 7faf70efb7c0 -1 Falling back to public interface 2024-09-11T05:07:33.675 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T05:07:33.673+0000 7f2cdbe70640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-11T05:07:33.680 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T05:07:33.677+0000 7f2cdbe70640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-11T05:07:36.766 INFO:tasks.ceph.osd.0.smithi138.stderr:2024-09-11T05:07:36.763+0000 7f2eceaa57c0 -1 osd.0 0 log_to_monitors true 2024-09-11T05:07:36.842 INFO:tasks.ceph.osd.2.smithi138.stderr:2024-09-11T05:07:36.839+0000 7f72688687c0 -1 osd.2 0 log_to_monitors true 2024-09-11T05:07:37.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2024-09-11T05:07:37.363+0000 7fe9ea20e7c0 -1 osd.1 0 log_to_monitors true 2024-09-11T05:07:37.369 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T05:07:37.365+0000 7ff6212007c0 -1 osd.4 0 log_to_monitors true 2024-09-11T05:07:37.459 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-09-11T05:07:37.457+0000 7f6f7decf7c0 -1 osd.5 0 log_to_monitors true 2024-09-11T05:07:37.817 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T05:07:37.813+0000 7faf70efb7c0 -1 osd.3 0 log_to_monitors true 2024-09-11T05:07:38.681 INFO:tasks.ceph.osd.0.smithi138.stderr:2024-09-11T05:07:38.679+0000 7f2ecaa4e640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T05:07:38.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2024-09-11T05:07:38.679+0000 7f7264811640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T05:07:38.687 INFO:tasks.ceph.osd.0.smithi138.stderr:2024-09-11T05:07:38.683+0000 7f2ec6888640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:07:38.692 INFO:tasks.ceph.osd.2.smithi138.stderr:2024-09-11T05:07:38.691+0000 7f726064b640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:07:38.692 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T05:07:38.689+0000 7ff61d1a9640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T05:07:38.698 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T05:07:38.697+0000 7ff618fe3640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:07:39.247 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T05:07:39.330 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-09-11T05:07:39.329+0000 7f6f79e78640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T05:07:39.333 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-09-11T05:07:39.329+0000 7f6f71caa640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:07:39.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2024-09-11T05:07:39.371+0000 7fe9e61b7640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T05:07:39.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2024-09-11T05:07:39.375+0000 7fe9ddfe9640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.375+0000 7f6740763640 10 monclient: get_monmap_and_config 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.375+0000 7f6740763640 10 monclient: build_initial_monmap 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.375+0000 7f6740763640 10 monclient: monmap: 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:07:39.380949+0000 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:07:39.380949+0000 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:39.379 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:39.380 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:39.380 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:39.380 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:39.380 INFO:teuthology.misc.health.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:39.380 INFO:teuthology.misc.health.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:39.380 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:39.382 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 1 Processor -- start 2024-09-11T05:07:39.382 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 1 -- start start 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient: init 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f673806fe60 0x7f67380a4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient(hunting): picked mon.noname-b con 0x7f673806fe60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f67380a3ad0 0x7f67380a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient(hunting): picked mon.noname-c con 0x7f67380a3ad0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67380a5a70 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67380a4f10 con 0x7f67380a5a70 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6738070240 con 0x7f67380a3ad0 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:39.383 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f67380a5430 con 0x7f673806fe60 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f6740763640 10 monclient(hunting): authenticate will time out at 942.816694s 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673ed00640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f67380a5a70 0x7f67380a5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:46594/0 (socket says 172.21.15.138:46594) 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673ed00640 1 -- 172.21.15.138:0/1968770147 learned_addr learned my addr 172.21.15.138:0/1968770147 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f673806fe60 0x7f67380a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient(hunting): get_auth_request con 0x7f673806fe60 auth_method 0 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:39.384 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673dcfe640 1 --2- 172.21.15.138:0/1968770147 >> v2:172.21.15.190:3300/0 conn(0x7f67380a3ad0 0x7f67380a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673dcfe640 10 monclient(hunting): get_auth_request con 0x7f67380a3ad0 auth_method 0 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673dcfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673dcfe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673dcfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient(hunting): handle_auth_done global_id 4175 payload 274 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 1 -- 172.21.15.138:0/1968770147 >> v2:172.21.15.190:3300/0 conn(0x7f67380a3ad0 msgr2=0x7f67380a49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 1 --2- 172.21.15.138:0/1968770147 >> v2:172.21.15.190:3300/0 conn(0x7f67380a3ad0 0x7f67380a49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 1 -- 172.21.15.138:0/1968770147 >> v1:172.21.15.138:6789/0 conn(0x7f67380a5a70 legacy=0x7f67380a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 1 monclient: found mon.noname-b 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:39.385 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.379+0000 7f673e4ff640 1 -- 172.21.15.138:0/1968770147 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67380a6d00 con 0x7f673806fe60 2024-09-11T05:07:39.389 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673e4ff640 1 -- 172.21.15.138:0/1968770147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f673806fe60 msgr2=0x7f67380a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:39.389 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673e4ff640 1 --2- 172.21.15.138:0/1968770147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f673806fe60 0x7f67380a4490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f67340097c0 tx=0x7f673401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=853d68b9f059c7fb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:39.389 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 1 -- 172.21.15.138:0/1968770147 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340b0070 con 0x7f673806fe60 2024-09-11T05:07:39.389 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:39.389 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:39.389 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: dump: 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: _finish_auth 0 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 1 -- 172.21.15.138:0/1968770147 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67340ab600 con 0x7f673806fe60 2024-09-11T05:07:39.390 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 1 -- 172.21.15.138:0/1968770147 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340ab9d0 con 0x7f673806fe60 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673d4fd640 10 monclient: dump: 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:39.391 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 5 monclient: authenticate success, global_id 4175 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 4 set_mon_vals no callback set 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient: shutdown 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- 172.21.15.138:0/1968770147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f673806fe60 msgr2=0x7f67380a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 --2- 172.21.15.138:0/1968770147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f673806fe60 0x7f67380a4490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f67340097c0 tx=0x7f673401c3f0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.392 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- 172.21.15.138:0/1968770147 shutdown_connections 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 --2- 172.21.15.138:0/1968770147 >> v2:172.21.15.190:3300/0 conn(0x7f67380a3ad0 0x7f67380a49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 --2- 172.21.15.138:0/1968770147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f673806fe60 0x7f67380a4490 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- 172.21.15.138:0/1968770147 >> 172.21.15.138:0/1968770147 conn(0x7f673806d190 msgr2=0x7f673806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- 172.21.15.138:0/1968770147 shutdown_connections 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- 172.21.15.138:0/1968770147 wait complete. 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient: build_initial_monmap 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient: monmap: 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:39.393 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 librados: starting msgr at 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 librados: starting objecter 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 Processor -- start 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- start start 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 librados: setting wanted keys 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 librados: calling monclient init 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient: init 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f673806fe60 0x7f6738125a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:39.394 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient(hunting): picked mon.noname-b con 0x7f673806fe60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f67380a3ad0 0x7f673813c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient(hunting): picked mon.noname-c con 0x7f67380a3ad0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67380a5a70 0x7f67381d0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient(hunting): picked mon.noname-a con 0x7f67380a5a70 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6738143640 con 0x7f67380a5a70 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f67381434c0 con 0x7f673806fe60 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f67381437c0 con 0x7f67380a3ad0 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f6740763640 10 monclient(hunting): authenticate will time out at 942.823515s 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f67380a3ad0 0x7f673813c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:39.395 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f67380a3ad0 0x7f673813c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:43002/0 (socket says 172.21.15.138:43002) 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 -- 172.21.15.138:0/4118483004 learned_addr learned my addr 172.21.15.138:0/4118483004 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient(hunting): get_auth_request con 0x7f67380a3ad0 auth_method 0 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67380a5a70 0x7f67381d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 10 monclient(hunting): get_auth_request con 0x7f67380a5a70 auth_method 0 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:39.396 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673e4ff640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f673806fe60 0x7f6738125a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 -- 172.21.15.138:0/4118483004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f673806fe60 msgr2=0x7f6738125a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f673806fe60 0x7f6738125a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 -- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67380a5a70 msgr2=0x7f67381d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67380a5a70 0x7f67381d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 monclient: found mon.noname-c 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:39.397 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673dcfe640 1 -- 172.21.15.138:0/4118483004 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6738126f50 con 0x7f67380a3ad0 2024-09-11T05:07:39.398 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673e4ff640 10 monclient: get_auth_request con 0x7f673806fe60 auth_method 0 2024-09-11T05:07:39.398 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673e4ff640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f673806fe60 0x7f6738125a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:39.398 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.383+0000 7f673ed00640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67380a5a70 0x7f67381d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f673dcfe640 1 -- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f67380a3ad0 msgr2=0x7f673813c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f673dcfe640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f67380a3ad0 0x7f673813c8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f672800d7d0 tx=0x7f672800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 client.?.objecter ms_handle_connect 0x7f67380a3ad0 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 1 -- 172.21.15.138:0/4118483004 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728004020 con 0x7f67380a3ad0 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: dump: 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:39.400 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: _finish_auth 0 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 1 -- 172.21.15.138:0/4118483004 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67280041c0 con 0x7f67380a3ad0 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 1 -- 172.21.15.138:0/4118483004 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67280043b0 con 0x7f67380a3ad0 2024-09-11T05:07:39.401 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 monclient: dump: 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 5 monclient: authenticate success, global_id 4178 2024-09-11T05:07:39.402 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f673ccfc640 4 set_mon_vals no callback set 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 10 monclient: _renew_subs 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673806e0f0 con 0x7f67380a3ad0 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 10 monclient: _renew_subs 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6738124ed0 con 0x7f67380a3ad0 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 1 librados: init done 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 1 -- 172.21.15.138:0/4118483004 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 4) ==== 48435+0+0 (secure 0 0 0) 0x7f6728013070 con 0x7f67380a3ad0 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 1 --2- 172.21.15.138:0/4118483004 >> v2:172.21.15.190:6800/1562782189 conn(0x7f671c03ce10 0x7f671c03f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67380a5ef0 con 0x7f67380a3ad0 2024-09-11T05:07:39.403 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 1 -- 172.21.15.138:0/4118483004 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1911+0+0 (secure 0 0 0) 0x7f672804e900 con 0x7f67380a3ad0 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 10 client.4178.objecter ms_dispatch 0x7f6738000ba0 osd_map(10..10 src has 1..10) 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f672effd640 20 client.4178.objecter dump_active .. 0 homeless 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f673e4ff640 1 --2- 172.21.15.138:0/4118483004 >> v2:172.21.15.190:6800/1562782189 conn(0x7f671c03ce10 0x7f671c03f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.387+0000 7f673e4ff640 10 monclient: get_auth_request con 0x7f671c03ce10 auth_method 0 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.391+0000 7f672effd640 10 client.4178.objecter ms_handle_connect 0x7f671c03ce10 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.391+0000 7f673e4ff640 1 --2- 172.21.15.138:0/4118483004 >> v2:172.21.15.190:6800/1562782189 conn(0x7f671c03ce10 0x7f671c03f2d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f67340ab000 tx=0x7f67340a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.395+0000 7f672effd640 1 -- 172.21.15.138:0/4118483004 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f6728003d30 con 0x7f67380a3ad0 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.395+0000 7f672effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:39.404 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.395+0000 7f672effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:39.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.563+0000 7f6740763640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:39.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.563+0000 7f6740763640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:39.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.563+0000 7f6740763640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:39.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.563+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f673806a5e0 con 0x7f67380a3ad0 2024-09-11T05:07:39.565 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.563+0000 7f672effd640 1 -- 172.21.15.138:0/4118483004 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+5606 (secure 0 0 0) 0x7f67280039a0 con 0x7f67380a3ad0 2024-09-11T05:07:39.566 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.563+0000 7f672effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:39.566 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.563+0000 7f672effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:39.566 INFO:teuthology.misc.health.smithi138.stdout: 2024-09-11T05:07:39.566 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":10,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","created":"2024-09-11T05:07:27.386635+0000","modified":"2024-09-11T05:07:38.672023+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T05:07:31.585387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"79670fd2-b5aa-408e-8aa4-18a8fb848972","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e96fe3a6-3058-4c40-aa87-26a89572cdfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a87dbce5-b78c-4070-90cc-d323fc444f4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e1a7ea3f-62f5-4d1f-b671-be183143eb20","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a1aca44b-3dce-4f17-bc06-69bdca9cba04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"87ad0149-d931-4ac6-8787-94f9a9609d86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:07:39.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 10 librados: watch_flush enter 2024-09-11T05:07:39.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 10 librados: watch_flush exit 2024-09-11T05:07:39.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 20 client.4178.objecter shutdown clearing up homeless session... 2024-09-11T05:07:39.568 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 10 client.4178.objecter successfully canceled tick 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 >> v2:172.21.15.190:6800/1562782189 conn(0x7f671c03ce10 msgr2=0x7f671c03f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 --2- 172.21.15.138:0/4118483004 >> v2:172.21.15.190:6800/1562782189 conn(0x7f671c03ce10 0x7f671c03f2d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f67340ab000 tx=0x7f67340a6000 comp rx=0 tx=0).stop 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 10 monclient: shutdown 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f67380a3ad0 msgr2=0x7f673813c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f67380a3ad0 0x7f673813c8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f672800d7d0 tx=0x7f672800dc90 comp rx=0 tx=0).stop 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 shutdown_connections 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 --2- 172.21.15.138:0/4118483004 >> v2:172.21.15.190:6800/1562782189 conn(0x7f671c03ce10 0x7f671c03f2d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f67380a5a70 0x7f67381d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f67380a3ad0 0x7f673813c8a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 --2- 172.21.15.138:0/4118483004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f673806fe60 0x7f6738125a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 >> 172.21.15.138:0/4118483004 conn(0x7f673806d190 msgr2=0x7f673813faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:39.569 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 shutdown_connections 2024-09-11T05:07:39.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 -- 172.21.15.138:0/4118483004 wait complete. 2024-09-11T05:07:39.570 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:39.567+0000 7f6740763640 1 librados: shutdown 2024-09-11T05:07:39.583 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-11T05:07:39.700 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T05:07:39.697+0000 7faf6cea4640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T05:07:39.707 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T05:07:39.705+0000 7faf68cde640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T05:07:46.189 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T05:07:46.310 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.306+0000 7fdb62866640 10 monclient: get_monmap_and_config 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.306+0000 7fdb62866640 10 monclient: build_initial_monmap 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.306+0000 7fdb62866640 10 monclient: monmap: 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:07:46.312361+0000 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:07:46.312361+0000 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:46.311 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:46.314 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 1 Processor -- start 2024-09-11T05:07:46.314 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 1 -- start start 2024-09-11T05:07:46.314 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient: init 2024-09-11T05:07:46.314 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:46.314 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-11T05:07:46.314 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c06fe60 0x7fdb5c0a4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient(hunting): picked mon.noname-b con 0x7fdb5c06fe60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdb5c0a3ad0 0x7fdb5c0a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient(hunting): picked mon.noname-c con 0x7fdb5c0a3ad0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb5c0a5a70 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb5c0a4f10 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdb5c070240 con 0x7fdb5c0a3ad0 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdb5c0a5430 con 0x7fdb5c06fe60 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb62866640 10 monclient(hunting): authenticate will time out at 949.749191s 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c06fe60 0x7fdb5c0a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.315 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c06fe60 0x7fdb5c0a4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:43018/0 (socket says 172.21.15.138:43018) 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 1 -- 172.21.15.138:0/1356281654 learned_addr learned my addr 172.21.15.138:0/1356281654 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request con 0x7fdb5c06fe60 auth_method 0 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5b7fe640 1 --2- 172.21.15.138:0/1356281654 >> v2:172.21.15.190:3300/0 conn(0x7fdb5c0a3ad0 0x7fdb5c0a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5b7fe640 10 monclient(hunting): get_auth_request con 0x7fdb5c0a3ad0 auth_method 0 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:46.316 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 1 -- 172.21.15.138:0/1356281654 >> v2:172.21.15.190:3300/0 conn(0x7fdb5c0a3ad0 msgr2=0x7fdb5c0a49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 1 --2- 172.21.15.138:0/1356281654 >> v2:172.21.15.190:3300/0 conn(0x7fdb5c0a3ad0 0x7fdb5c0a49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 1 -- 172.21.15.138:0/1356281654 >> v1:172.21.15.190:6789/0 conn(0x7fdb5c0a5a70 legacy=0x7fdb5c0a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 1 monclient: found mon.noname-b 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:46.317 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.310+0000 7fdb5bfff640 1 -- 172.21.15.138:0/1356281654 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c0a6d00 con 0x7fdb5c06fe60 2024-09-11T05:07:46.318 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.314+0000 7fdb5bfff640 1 -- 172.21.15.138:0/1356281654 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c06fe60 msgr2=0x7fdb5c0a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:46.318 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.314+0000 7fdb5bfff640 1 --2- 172.21.15.138:0/1356281654 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c06fe60 0x7fdb5c0a4490 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009870 tx=0x7fdb4c01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e5e6dbf199c134 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:46.320 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 1 -- 172.21.15.138:0/1356281654 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0b0070 con 0x7fdb5c06fe60 2024-09-11T05:07:46.321 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:46.321 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:46.321 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: dump: 2024-09-11T05:07:46.321 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:46.321 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.321 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.321 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.321 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: _finish_auth 0 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 1 -- 172.21.15.138:0/1356281654 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4c0ab600 con 0x7fdb5c06fe60 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 1 -- 172.21.15.138:0/1356281654 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0ab9d0 con 0x7fdb5c06fe60 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:46.322 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:46.323 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5affd640 10 monclient: dump: 2024-09-11T05:07:46.323 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:46.323 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.323 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.323 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.324 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:46.324 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.324 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:46.324 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:46.324 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:46.324 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:46.324 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 5 monclient: authenticate success, global_id 4193 2024-09-11T05:07:46.325 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:46.325 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 4 set_mon_vals no callback set 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient: shutdown 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- 172.21.15.138:0/1356281654 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c06fe60 msgr2=0x7fdb5c0a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 --2- 172.21.15.138:0/1356281654 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c06fe60 0x7fdb5c0a4490 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009870 tx=0x7fdb4c01c4b0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- 172.21.15.138:0/1356281654 shutdown_connections 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 --2- 172.21.15.138:0/1356281654 >> v2:172.21.15.190:3300/0 conn(0x7fdb5c0a3ad0 0x7fdb5c0a49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 --2- 172.21.15.138:0/1356281654 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c06fe60 0x7fdb5c0a4490 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- 172.21.15.138:0/1356281654 >> 172.21.15.138:0/1356281654 conn(0x7fdb5c06d190 msgr2=0x7fdb5c06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:46.330 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- 172.21.15.138:0/1356281654 shutdown_connections 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- 172.21.15.138:0/1356281654 wait complete. 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient: build_initial_monmap 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient: monmap: 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:epoch 0 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:created 0.000000 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:46.331 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 librados: starting msgr at 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 librados: starting objecter 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 Processor -- start 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- start start 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 librados: setting wanted keys 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 librados: calling monclient init 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient: init 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb5c06fe60 0x7fdb5c128c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient(hunting): picked mon.noname-a con 0x7fdb5c06fe60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb5c0a3ad0 0x7fdb5c126b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient(hunting): picked mon.noname-b con 0x7fdb5c0a3ad0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:46.332 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c0a5a70 0x7fdb5c13faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient(hunting): picked mon.noname-c con 0x7fdb5c0a5a70 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c143640 con 0x7fdb5c06fe60 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c1434c0 con 0x7fdb5c0a3ad0 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdb5c1437c0 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb62866640 10 monclient(hunting): authenticate will time out at 949.759810s 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb60e03640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c0a5a70 0x7fdb5c13faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb5c06fe60 0x7fdb5c128c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb60e03640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c0a5a70 0x7fdb5c13faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:43024/0 (socket says 172.21.15.138:43024) 2024-09-11T05:07:46.333 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb60e03640 1 -- 172.21.15.138:0/3391752995 learned_addr learned my addr 172.21.15.138:0/3391752995 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb60e03640 10 monclient(hunting): get_auth_request con 0x7fdb5c0a5a70 auth_method 0 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb60e03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb60e03640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb60e03640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request con 0x7fdb5c06fe60 auth_method 0 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.318+0000 7fdb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb5b7fe640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb5c0a3ad0 0x7fdb5c126b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.334 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 1 -- 172.21.15.138:0/3391752995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb5c0a3ad0 msgr2=0x7fdb5c126b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb5c0a3ad0 0x7fdb5c126b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 1 -- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb5c06fe60 msgr2=0x7fdb5c128c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb5c06fe60 0x7fdb5c128c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 10 monclient: _finish_hunting 0 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 1 monclient: found mon.noname-c 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 1 -- 172.21.15.138:0/3391752995 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c128120 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb5bfff640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb5c06fe60 0x7fdb5c128c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb5b7fe640 10 monclient: get_auth_request con 0x7fdb5c0a3ad0 auth_method 0 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb5b7fe640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb5c0a3ad0 0x7fdb5c126b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 1 -- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c0a5a70 msgr2=0x7fdb5c13faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb60e03640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c0a5a70 0x7fdb5c13faf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c780 tx=0x7fdb5400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:46.335 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 client.?.objecter ms_handle_connect 0x7fdb5c0a5a70 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 1 -- 172.21.15.138:0/3391752995 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400ac60 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: dump: 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:46.336 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: _finish_auth 0 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 1 -- 172.21.15.138:0/3391752995 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb5400ae00 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 5 monclient: authenticate success, global_id 4196 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 10 monclient: _renew_subs 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb5c140790 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 1 -- 172.21.15.138:0/3391752995 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54004bb0 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:46.337 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 10 monclient: dump: 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:epoch 1 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr: 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 10 monclient: _renew_subs 2024-09-11T05:07:46.338 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb5a7fc640 4 set_mon_vals no callback set 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb5c129ec0 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 1 librados: init done 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb5c0a5ef0 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.322+0000 7fdb58ff9640 1 -- 172.21.15.138:0/3391752995 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdb54006b60 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb58ff9640 1 --2- 172.21.15.138:0/3391752995 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4803cea0 0x7fdb4803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb58ff9640 1 -- 172.21.15.138:0/3391752995 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (secure 0 0 0) 0x7fdb5404f480 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb58ff9640 10 client.4196.objecter ms_dispatch 0x7fdb5c000ba0 osd_map(15..15 src has 1..15) 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb58ff9640 3 client.4196.objecter handle_osd_map got epochs [15,15] > 0 2024-09-11T05:07:46.339 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb58ff9640 3 client.4196.objecter handle_osd_map decoding full epoch 15 2024-09-11T05:07:46.340 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb58ff9640 20 client.4196.objecter dump_active .. 0 homeless 2024-09-11T05:07:46.340 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb5bfff640 1 --2- 172.21.15.138:0/3391752995 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4803cea0 0x7fdb4803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.340 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb5bfff640 10 monclient: get_auth_request con 0x7fdb4803cea0 auth_method 0 2024-09-11T05:07:46.340 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb5bfff640 1 --2- 172.21.15.138:0/3391752995 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4803cea0 0x7fdb4803f360 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0ab000 tx=0x7fdb4c0a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:46.340 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.326+0000 7fdb58ff9640 10 client.4196.objecter ms_handle_connect 0x7fdb4803cea0 2024-09-11T05:07:46.341 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.330+0000 7fdb58ff9640 1 -- 172.21.15.138:0/3391752995 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fdb54020070 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.341 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.330+0000 7fdb58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:46.341 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.330+0000 7fdb58ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:46.500 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.498+0000 7fdb62866640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:46.500 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.498+0000 7fdb62866640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:46.500 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.498+0000 7fdb62866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:46.500 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.498+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb5c12a260 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.501 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.498+0000 7fdb58ff9640 1 -- 172.21.15.138:0/3391752995 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+9041 (secure 0 0 0) 0x7fdb54004e10 con 0x7fdb5c0a5a70 2024-09-11T05:07:46.501 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.498+0000 7fdb58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:46.501 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.498+0000 7fdb58ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:46.501 INFO:teuthology.misc.health.smithi138.stdout: 2024-09-11T05:07:46.502 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":15,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","created":"2024-09-11T05:07:27.386635+0000","modified":"2024-09-11T05:07:44.483812+0000","last_up_change":"2024-09-11T05:07:40.695452+0000","last_in_change":"2024-09-11T05:07:31.585387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:07:40.718263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79670fd2-b5aa-408e-8aa4-18a8fb848972","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":16171034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":16171034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":16171034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":16171034}]},"public_addr":"172.21.15.138:6800/16171034","cluster_addr":"172.21.15.138:6801/16171034","heartbeat_back_addr":"172.21.15.138:6803/16171034","heartbeat_front_addr":"172.21.15.138:6802/16171034","state":["exists","up"]},{"osd":1,"uuid":"e96fe3a6-3058-4c40-aa87-26a89572cdfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":4002974512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4002974512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":4002974512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":4002974512}]},"public_addr":"172.21.15.138:6808/4002974512","cluster_addr":"172.21.15.138:6809/4002974512","heartbeat_back_addr":"172.21.15.138:6811/4002974512","heartbeat_front_addr":"172.21.15.138:6810/4002974512","state":["exists","up"]},{"osd":2,"uuid":"a87dbce5-b78c-4070-90cc-d323fc444f4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":81047329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":81047329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":81047329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":81047329}]},"public_addr":"172.21.15.138:6804/81047329","cluster_addr":"172.21.15.138:6805/81047329","heartbeat_back_addr":"172.21.15.138:6807/81047329","heartbeat_front_addr":"172.21.15.138:6806/81047329","state":["exists","up"]},{"osd":3,"uuid":"e1a7ea3f-62f5-4d1f-b671-be183143eb20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3046884208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3046884208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3046884208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3046884208}]},"public_addr":"172.21.15.190:6809/3046884208","cluster_addr":"172.21.15.190:6810/3046884208","heartbeat_back_addr":"172.21.15.190:6812/3046884208","heartbeat_front_addr":"172.21.15.190:6811/3046884208","state":["exists","up"]},{"osd":4,"uuid":"a1aca44b-3dce-4f17-bc06-69bdca9cba04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":1118550024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1118550024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1118550024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":1118550024}]},"public_addr":"172.21.15.190:6801/1118550024","cluster_addr":"172.21.15.190:6802/1118550024","heartbeat_back_addr":"172.21.15.190:6804/1118550024","heartbeat_front_addr":"172.21.15.190:6803/1118550024","state":["exists","up"]},{"osd":5,"uuid":"87ad0149-d931-4ac6-8787-94f9a9609d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":189369151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":189369151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":189369151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":189369151}]},"public_addr":"172.21.15.190:6805/189369151","cluster_addr":"172.21.15.190:6806/189369151","heartbeat_back_addr":"172.21.15.190:6808/189369151","heartbeat_front_addr":"172.21.15.190:6807/189369151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.795496+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.359385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.802420+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.791133+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.377964+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.428799+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:07:46.503 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.498+0000 7fdb62866640 10 librados: watch_flush enter 2024-09-11T05:07:46.503 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 10 librados: watch_flush exit 2024-09-11T05:07:46.503 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 20 client.4196.objecter shutdown clearing up homeless session... 2024-09-11T05:07:46.503 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 10 client.4196.objecter successfully canceled tick 2024-09-11T05:07:46.504 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4803cea0 msgr2=0x7fdb4803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:46.504 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 --2- 172.21.15.138:0/3391752995 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4803cea0 0x7fdb4803f360 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0ab000 tx=0x7fdb4c0a6000 comp rx=0 tx=0).stop 2024-09-11T05:07:46.504 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 10 monclient: shutdown 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c0a5a70 msgr2=0x7fdb5c13faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c0a5a70 0x7fdb5c13faf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400c780 tx=0x7fdb5400cc40 comp rx=0 tx=0).stop 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 shutdown_connections 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 --2- 172.21.15.138:0/3391752995 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4803cea0 0x7fdb4803f360 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb5c0a5a70 0x7fdb5c13faf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb5c0a3ad0 0x7fdb5c126b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 --2- 172.21.15.138:0/3391752995 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb5c06fe60 0x7fdb5c128c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 >> 172.21.15.138:0/3391752995 conn(0x7fdb5c06d190 msgr2=0x7fdb5c09f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 shutdown_connections 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 -- 172.21.15.138:0/3391752995 wait complete. 2024-09-11T05:07:46.505 INFO:teuthology.misc.health.smithi138.stderr:2024-09-11T05:07:46.502+0000 7fdb62866640 1 librados: shutdown 2024-09-11T05:07:46.519 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-11T05:07:46.519 INFO:tasks.ceph:Creating RBD pool 2024-09-11T05:07:46.519 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T05:07:46.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.630+0000 7fad2088f640 10 monclient: get_monmap_and_config 2024-09-11T05:07:46.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.630+0000 7fad2088f640 10 monclient: build_initial_monmap 2024-09-11T05:07:46.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.630+0000 7fad2088f640 10 monclient: monmap: 2024-09-11T05:07:46.635 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:46.637143+0000 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:46.637143+0000 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:46.636 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:46.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 1 Processor -- start 2024-09-11T05:07:46.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 1 -- start start 2024-09-11T05:07:46.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient: init 2024-09-11T05:07:46.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:46.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fad1c09da20 0x7fad1c0a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient(hunting): picked mon.noname-a con 0x7fad1c09da20 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad1c0a1f40 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad1c0a23d0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad1c0a2820 con 0x7fad1c0a1f40 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad1c0a2d90 con 0x7fad1c0a23d0 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fad1c0a3070 con 0x7fad1c09da20 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad2088f640 10 monclient(hunting): authenticate will time out at 950.073364s 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fad1c09da20 0x7fad1c0a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad19d9b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fad1c0a1f40 0x7fad1c136170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:46604/0 (socket says 172.21.15.138:46604) 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 1 --2- 172.21.15.138:0/94702113 >> v2:172.21.15.138:3300/0 conn(0x7fad1c09da20 0x7fad1c0a1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:55052/0 (socket says 172.21.15.138:55052) 2024-09-11T05:07:46.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad19d9b640 1 -- 172.21.15.138:0/94702113 learned_addr learned my addr 172.21.15.138:0/94702113 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:46.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient(hunting): get_auth_request con 0x7fad1c09da20 auth_method 0 2024-09-11T05:07:46.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:46.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:46.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:46.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:46.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:46.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:46.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient(hunting): handle_auth_done global_id 4188 payload 274 2024-09-11T05:07:46.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 1 -- 172.21.15.138:0/94702113 >> v1:172.21.15.190:6789/0 conn(0x7fad1c0a23d0 legacy=0x7fad1c136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:46.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 1 -- 172.21.15.138:0/94702113 >> v1:172.21.15.138:6789/0 conn(0x7fad1c0a1f40 legacy=0x7fad1c136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:46.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient: _finish_hunting 0 2024-09-11T05:07:46.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 1 monclient: found mon.noname-a 2024-09-11T05:07:46.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:46.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:46.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.634+0000 7fad1a59c640 1 -- 172.21.15.138:0/94702113 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c0a3350 con 0x7fad1c09da20 2024-09-11T05:07:46.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1a59c640 1 -- 172.21.15.138:0/94702113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 msgr2=0x7fad1c0a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1a59c640 1 --2- 172.21.15.138:0/94702113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c0a1a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0097c0 tx=0x7fad0c01cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e663ad9f9aceea0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 1 -- 172.21.15.138:0/94702113 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0b0070 con 0x7fad1c09da20 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: dump: 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:46.643 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: _finish_auth 0 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: _check_auth_tickets 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 1 -- 172.21.15.138:0/94702113 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0c0ab600 con 0x7fad1c09da20 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 5 monclient: authenticate success, global_id 4188 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 1 -- 172.21.15.138:0/94702113 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0ab9d0 con 0x7fad1c09da20 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 4 set_mon_vals no callback set 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:46.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad1959a640 10 monclient: dump: 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 10 monclient: shutdown 2024-09-11T05:07:46.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 -- 172.21.15.138:0/94702113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 msgr2=0x7fad1c0a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 --2- 172.21.15.138:0/94702113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c0a1a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0097c0 tx=0x7fad0c01cdc0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 -- 172.21.15.138:0/94702113 shutdown_connections 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 --2- 172.21.15.138:0/94702113 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c0a1a00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 -- 172.21.15.138:0/94702113 >> 172.21.15.138:0/94702113 conn(0x7fad1c0a3b00 msgr2=0x7fad1c0a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 -- 172.21.15.138:0/94702113 shutdown_connections 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 -- 172.21.15.138:0/94702113 wait complete. 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 10 monclient: build_initial_monmap 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 10 monclient: monmap: 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:46.646 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.647 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:46.647 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:46.647 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:46.647 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 librados: starting msgr at 2024-09-11T05:07:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 librados: starting objecter 2024-09-11T05:07:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 Processor -- start 2024-09-11T05:07:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 -- start start 2024-09-11T05:07:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 librados: setting wanted keys 2024-09-11T05:07:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 1 librados: calling monclient init 2024-09-11T05:07:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.638+0000 7fad2088f640 10 monclient: init 2024-09-11T05:07:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c1ce480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient(hunting): picked mon.noname-a con 0x7fad1c09da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fad1c0a1f40 0x7fad1c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient(hunting): picked mon.noname-c con 0x7fad1c0a1f40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1c0a23d0 0x7fad1c13eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient(hunting): picked mon.noname-b con 0x7fad1c0a23d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c141ec0 con 0x7fad1c09da20 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c141d40 con 0x7fad1c0a23d0 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fad1c142040 con 0x7fad1c0a1f40 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient(hunting): authenticate will time out at 950.080444s 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c1ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c1ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:55062/0 (socket says 172.21.15.138:55062) 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 -- 172.21.15.138:0/203377024 learned_addr learned my addr 172.21.15.138:0/203377024 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient(hunting): get_auth_request con 0x7fad1c09da20 auth_method 0 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad19d9b640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fad1c0a1f40 0x7fad1c13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:46.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad19d9b640 10 monclient(hunting): get_auth_request con 0x7fad1c0a1f40 auth_method 0 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad19d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad19d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad19d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad19d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient(hunting): handle_auth_done global_id 4191 payload 824 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 -- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fad1c0a1f40 msgr2=0x7fad1c13e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fad1c0a1f40 0x7fad1c13e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1ad9d640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1c0a23d0 0x7fad1c13eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 -- 172.21.15.138:0/203377024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1c0a23d0 msgr2=0x7fad1c13eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1c0a23d0 0x7fad1c13eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient: _finish_hunting 0 2024-09-11T05:07:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 monclient: found mon.noname-a 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 -- 172.21.15.138:0/203377024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c13ad40 con 0x7fad1c09da20 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 -- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 msgr2=0x7fad1c1ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad1a59c640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c1ce480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0ae040 tx=0x7fad0c01aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 client.?.objecter ms_handle_connect 0x7fad1c09da20 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 1 -- 172.21.15.138:0/203377024 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0b0050 con 0x7fad1c09da20 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: dump: 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: _finish_auth 0 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 1 -- 172.21.15.138:0/203377024 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0c0bc070 con 0x7fad1c09da20 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 5 monclient: authenticate success, global_id 4191 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient: _renew_subs 2024-09-11T05:07:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1c13b020 con 0x7fad1c09da20 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 1 -- 172.21.15.138:0/203377024 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0b5c80 con 0x7fad1c09da20 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad0affd640 10 monclient: dump: 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:46.657 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient: _renew_subs 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad1c13eff0 con 0x7fad1c09da20 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad18d99640 4 set_mon_vals no callback set 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.642+0000 7fad2088f640 1 librados: init done 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad2088f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad2088f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad2088f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad1c0a1a00 con 0x7fad1c09da20 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad0affd640 1 -- 172.21.15.138:0/203377024 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fad0c0c1460 con 0x7fad1c09da20 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad0affd640 1 --2- 172.21.15.138:0/203377024 >> v2:172.21.15.190:6800/1562782189 conn(0x7facfc03cea0 0x7facfc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad0affd640 1 -- 172.21.15.138:0/203377024 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (secure 0 0 0) 0x7fad0c0e9ab0 con 0x7fad1c09da20 2024-09-11T05:07:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad0affd640 10 client.4191.objecter ms_dispatch 0x7fad1c000ba0 osd_map(15..15 src has 1..15) 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad0affd640 3 client.4191.objecter handle_osd_map got epochs [15,15] > 0 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad0affd640 3 client.4191.objecter handle_osd_map decoding full epoch 15 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad0affd640 20 client.4191.objecter dump_active .. 0 homeless 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad19d9b640 1 --2- 172.21.15.138:0/203377024 >> v2:172.21.15.190:6800/1562782189 conn(0x7facfc03cea0 0x7facfc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad19d9b640 10 monclient: get_auth_request con 0x7facfc03cea0 auth_method 0 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad0affd640 10 client.4191.objecter ms_handle_connect 0x7facfc03cea0 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.646+0000 7fad19d9b640 1 --2- 172.21.15.138:0/203377024 >> v2:172.21.15.190:6800/1562782189 conn(0x7facfc03cea0 0x7facfc03f360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fad10007ca0 tx=0x7fad10007bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.650+0000 7fad0affd640 1 -- 172.21.15.138:0/203377024 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fad0c016070 con 0x7fad1c09da20 2024-09-11T05:07:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.650+0000 7fad0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.650+0000 7fad0affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:46.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.818+0000 7fad2088f640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T05:07:46.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.818+0000 7fad2088f640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T05:07:46.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.818+0000 7fad2088f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:46.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:46.818+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fad1c06cd10 con 0x7fad1c09da20 2024-09-11T05:07:47.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.506+0000 7fad0affd640 1 -- 172.21.15.138:0/203377024 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (secure 0 0 0) 0x7fad0c0ab4c0 con 0x7fad1c09da20 2024-09-11T05:07:47.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.506+0000 7fad0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T05:07:47.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.506+0000 7fad0affd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-11T05:07:47.512 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'rbd' created 2024-09-11T05:07:47.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 10 librados: watch_flush enter 2024-09-11T05:07:47.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 10 librados: watch_flush exit 2024-09-11T05:07:47.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 20 client.4191.objecter shutdown clearing up homeless session... 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 10 client.4191.objecter successfully canceled tick 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 >> v2:172.21.15.190:6800/1562782189 conn(0x7facfc03cea0 msgr2=0x7facfc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 --2- 172.21.15.138:0/203377024 >> v2:172.21.15.190:6800/1562782189 conn(0x7facfc03cea0 0x7facfc03f360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fad10007ca0 tx=0x7fad10007bf0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 10 monclient: shutdown 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 msgr2=0x7fad1c1ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c1ce480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0ae040 tx=0x7fad0c01aba0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 shutdown_connections 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 --2- 172.21.15.138:0/203377024 >> v2:172.21.15.190:6800/1562782189 conn(0x7facfc03cea0 0x7facfc03f360 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad1c0a23d0 0x7fad1c13eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fad1c0a1f40 0x7fad1c13e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 --2- 172.21.15.138:0/203377024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fad1c09da20 0x7fad1c1ce480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 >> 172.21.15.138:0/203377024 conn(0x7fad1c0a3b00 msgr2=0x7fad1c136010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:47.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 shutdown_connections 2024-09-11T05:07:47.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.510+0000 7fad2088f640 1 -- 172.21.15.138:0/203377024 wait complete. 2024-09-11T05:07:47.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.514+0000 7fad2088f640 1 librados: shutdown 2024-09-11T05:07:47.535 DEBUG:teuthology.orchestra.run.smithi138:> rbd --cluster ceph pool init rbd 2024-09-11T05:07:47.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.590+0000 7f4fb34f6100 10 monclient: get_monmap_and_config 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.590+0000 7f4fb34f6100 10 monclient: build_initial_monmap 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.590+0000 7f4fb34f6100 10 monclient: monmap: 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:47.596405+0000 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:47.596405+0000 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:47.595 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:47.596 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:47.596 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:47.596 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:47.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.590+0000 7f4fb34f6100 1 Processor -- start 2024-09-11T05:07:47.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.590+0000 7f4fb34f6100 1 -- start start 2024-09-11T05:07:47.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.590+0000 7f4fb34f6100 10 monclient: init 2024-09-11T05:07:47.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient: _reopen_session rank -1 2024-09-11T05:07:47.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 1 --2- >> v2:172.21.15.138:3300/0 conn(0x55d3f2e4c4e0 0x55d3f2e4c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient(hunting): picked mon.noname-a con 0x55d3f2e4c4e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 1 --2- >> v2:172.21.15.190:3300/0 conn(0x55d3f2e4ce00 0x55d3f2ee5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient(hunting): picked mon.noname-c con 0x55d3f2e4ce00 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient(hunting): picked mon.noname-a-legacy con 0x55d3f2ee5fd0 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55d3f2c96c80 con 0x55d3f2ee5fd0 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x55d3f2d18160 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x55d3f2d28d30 con 0x55d3f2e4ce00 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient(hunting): _renew_subs 2024-09-11T05:07:47.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb34f6100 10 monclient(hunting): authenticate will time out at 951.032519s 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x55d3f2e4c4e0 0x55d3f2e4c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb3261640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x55d3f2ee5fd0 0x55d3f2eecef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38354/0 (socket says 172.21.15.138:38354) 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb3261640 1 -- 172.21.15.138:0/790653930 learned_addr learned my addr 172.21.15.138:0/790653930 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient(hunting): get_auth_request con 0x55d3f2e4c4e0 auth_method 0 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb17fe640 1 --2- 172.21.15.138:0/790653930 >> v2:172.21.15.190:3300/0 conn(0x55d3f2e4ce00 0x55d3f2ee5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb17fe640 10 monclient(hunting): get_auth_request con 0x55d3f2e4ce00 auth_method 0 2024-09-11T05:07:47.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb17fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb17fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb17fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 1 -- 172.21.15.138:0/790653930 >> v2:172.21.15.190:3300/0 conn(0x55d3f2e4ce00 msgr2=0x55d3f2ee5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 1 --2- 172.21.15.138:0/790653930 >> v2:172.21.15.190:3300/0 conn(0x55d3f2e4ce00 0x55d3f2ee5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 1 -- 172.21.15.138:0/790653930 >> v1:172.21.15.138:6789/0 conn(0x55d3f2ee5fd0 legacy=0x55d3f2eecef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 1 monclient: found mon.noname-a 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:47.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.594+0000 7f4fb1fff640 1 -- 172.21.15.138:0/790653930 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3f2e48490 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb1fff640 1 -- 172.21.15.138:0/790653930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 msgr2=0x55d3f2e4c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb1fff640 1 --2- 172.21.15.138:0/790653930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 0x55d3f2e4c8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400a4a0 tx=0x7f4fa401af80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30eb2caa7c0027b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 1 -- 172.21.15.138:0/790653930 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40b0070 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: dump: 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 5 monclient: authenticate success, global_id 4194 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 1 -- 172.21.15.138:0/790653930 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa40a3cf0 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 10 monclient: get_monmap_and_config success 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 4 set_mon_vals no callback set 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 1 -- 172.21.15.138:0/790653930 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40ab640 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb0ffd640 10 monclient: dump: 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:47.607 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 10 monclient: shutdown 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 -- 172.21.15.138:0/790653930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 msgr2=0x55d3f2e4c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 --2- 172.21.15.138:0/790653930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 0x55d3f2e4c8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400a4a0 tx=0x7f4fa401af80 comp rx=0 tx=0).stop 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 -- 172.21.15.138:0/790653930 shutdown_connections 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 --2- 172.21.15.138:0/790653930 >> v2:172.21.15.190:3300/0 conn(0x55d3f2e4ce00 0x55d3f2ee5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 --2- 172.21.15.138:0/790653930 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 0x55d3f2e4c8c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 -- 172.21.15.138:0/790653930 >> 172.21.15.138:0/790653930 conn(0x55d3f2e4d5d0 msgr2=0x55d3f2e4d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 -- 172.21.15.138:0/790653930 shutdown_connections 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 -- 172.21.15.138:0/790653930 wait complete. 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 10 monclient: build_initial_monmap 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 10 monclient: monmap: 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 librados: starting msgr at 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 librados: starting objecter 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 Processor -- start 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 -- start start 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 librados: setting wanted keys 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 1 librados: calling monclient init 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.598+0000 7f4fb34f6100 10 monclient: init 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient: _reopen_session rank -1 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 0x55d3f2eed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient(hunting): picked mon.noname-a con 0x55d3f2e4c4e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x55d3f2e4ce00 0x55d3f2ee4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient(hunting): picked mon.noname-c con 0x55d3f2e4ce00 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3f2ee5fd0 0x55d3f2ee54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient(hunting): picked mon.noname-b con 0x55d3f2ee5fd0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x55d3f2cbda70 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d3f2d1bf70 con 0x55d3f2ee5fd0 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x55d3f2e498f0 con 0x55d3f2e4ce00 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient(hunting): _renew_subs 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient(hunting): authenticate will time out at 951.040161s 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 0x55d3f2eed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x55d3f2e4ce00 0x55d3f2ee4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 0x55d3f2eed6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46156/0 (socket says 172.21.15.138:46156) 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 -- 172.21.15.138:0/2027105893 learned_addr learned my addr 172.21.15.138:0/2027105893 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 10 monclient(hunting): get_auth_request con 0x55d3f2e4ce00 auth_method 0 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient(hunting): get_auth_request con 0x55d3f2e4c4e0 auth_method 0 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb3261640 1 --2- 172.21.15.138:0/2027105893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3f2ee5fd0 0x55d3f2ee54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb3261640 10 monclient(hunting): get_auth_request con 0x55d3f2ee5fd0 auth_method 0 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb3261640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb3261640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb3261640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 -- 172.21.15.138:0/2027105893 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x55d3f2e4ce00 msgr2=0x55d3f2ee4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 --2- 172.21.15.138:0/2027105893 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x55d3f2e4ce00 0x55d3f2ee4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 -- 172.21.15.138:0/2027105893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3f2ee5fd0 msgr2=0x55d3f2ee54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 --2- 172.21.15.138:0/2027105893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d3f2ee5fd0 0x55d3f2ee54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:47.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:47.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 monclient: found mon.noname-a 2024-09-11T05:07:47.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:47.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:47.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3f2eeabc0 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb17fe640 1 --2- 172.21.15.138:0/2027105893 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x55d3f2e4ce00 0x55d3f2ee4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:47.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 -- 172.21.15.138:0/2027105893 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 msgr2=0x55d3f2eed6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:47.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb1fff640 1 --2- 172.21.15.138:0/2027105893 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 0x55d3f2eed6e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40ae040 tx=0x7f4fa4017d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:47.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 client.?.objecter ms_handle_connect 0x55d3f2e4c4e0 2024-09-11T05:07:47.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:47.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40b0040 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: dump: 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:47.617 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa40a3af0 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa40b37a0 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fa9ffb640 10 monclient: dump: 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:47.618 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fabfff640 4 set_mon_vals no callback set 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 5 monclient: authenticate success, global_id 4197 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient: _renew_subs 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:47.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d3f2eec680 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:47.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient: _renew_subs 2024-09-11T05:07:47.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:47.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3f2ee59e0 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 1 librados: init done 2024-09-11T05:07:47.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.602+0000 7f4fb34f6100 10 librados: wait_for_osdmap waiting 2024-09-11T05:07:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f4fa40b3070 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fa9ffb640 1 --2- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f9803afd0 0x7f4f9803d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3495+0+0 (secure 0 0 0) 0x7f4fa40e7e10 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fa9ffb640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_map(16..16 src has 1..16) 2024-09-11T05:07:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fa9ffb640 3 client.4197.objecter handle_osd_map got epochs [16,16] > 0 2024-09-11T05:07:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fa9ffb640 3 client.4197.objecter handle_osd_map decoding full epoch 16 2024-09-11T05:07:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fa9ffb640 20 client.4197.objecter dump_active .. 0 homeless 2024-09-11T05:07:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fb34f6100 10 librados: wait_for_osdmap done waiting 2024-09-11T05:07:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fb34f6100 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T05:07:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fb34f6100 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T05:07:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fb34f6100 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55d3f2ee9100 con 0x55d3f2e4c4e0 2024-09-11T05:07:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fb17fe640 1 --2- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f9803afd0 0x7f4f9803d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.606+0000 7f4fb17fe640 10 monclient: get_auth_request con 0x7f4f9803afd0 auth_method 0 2024-09-11T05:07:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.610+0000 7f4fa9ffb640 10 client.4197.objecter ms_handle_connect 0x7f4f9803afd0 2024-09-11T05:07:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:47.610+0000 7f4fb17fe640 1 --2- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f9803afd0 0x7f4f9803d490 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x55d3f2eee110 tx=0x7f4fa0005b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:48.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.510+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (secure 0 0 0) 0x55d3f2e48490 con 0x55d3f2e4c4e0 2024-09-11T05:07:48.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.510+0000 7f4fa9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T05:07:48.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.510+0000 7f4fa9ffb640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-11T05:07:48.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.510+0000 7f4fb34f6100 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:48.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.510+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4fa40a3cf0 con 0x55d3f2e4c4e0 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.514+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f4fa40bdb20 con 0x55d3f2e4c4e0 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.514+0000 7f4fa9ffb640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.514+0000 7f4fabfff640 10 client.4197.objecter _get_latest_version latest 17, waiting 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.514+0000 7f4fabfff640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.514+0000 7f4fabfff640 10 monclient: _renew_subs 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.514+0000 7f4fabfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.514+0000 7f4fabfff640 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=17}) -- 0x7f4f8c003ca0 con 0x55d3f2e4c4e0 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 8 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (secure 0 0 0) 0x7f4fa40bcc10 con 0x55d3f2e4c4e0 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fa9ffb640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_map(17..17 src has 1..17) 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fa9ffb640 3 client.4197.objecter handle_osd_map got epochs [17,17] > 16 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fa9ffb640 3 client.4197.objecter handle_osd_map decoding incremental epoch 17 2024-09-11T05:07:48.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fa9ffb640 20 client.4197.objecter dump_active .. 0 homeless 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 10 librados: create oid=rbd_trash nspace= 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 10 client.4197.objecter _op_submit op 0x55d3f2ee9620 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 20 client.4197.objecter _calc_target epoch 17 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 20 client.4197.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 10 client.4197.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 1 --2- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6805/189369151 conn(0x55d3f2eff0f0 0x55d3f2f014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 20 client.4197.objecter _get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 10 client.4197.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 20 client.4197.objecter get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 15 client.4197.objecter _session_op_assign 5 1 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 15 client.4197.objecter _send_op 1 to 2.3 on osd.5 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 --> v2:172.21.15.190:6805/189369151 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x55d3f2f01a20 con 0x55d3f2eff0f0 2024-09-11T05:07:48.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:48.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb34f6100 5 client.4197.objecter 1 in flight 2024-09-11T05:07:48.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb3261640 1 --2- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6805/189369151 conn(0x55d3f2eff0f0 0x55d3f2f014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:48.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.518+0000 7f4fb3261640 10 monclient: get_auth_request con 0x55d3f2eff0f0 auth_method 0 2024-09-11T05:07:48.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.522+0000 7f4fa9ffb640 10 client.4197.objecter ms_handle_connect 0x55d3f2eff0f0 2024-09-11T05:07:48.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.522+0000 7f4fb3261640 1 --2- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6805/189369151 conn(0x55d3f2eff0f0 0x55d3f2f014e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:48.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 1 -- 172.21.15.138:0/2027105893 <== osd.5 v2:172.21.15.190:6805/189369151 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f4f9c008040 con 0x55d3f2eff0f0 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 10 client.4197.objecter in handle_osd_op_reply 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 7 client.4197.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 10 client.4197.objecter op 0 rval 0 len 0 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 15 client.4197.objecter handle_osd_op_reply completed tid 1 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 15 client.4197.objecter _finish_op 1 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 15 client.4197.objecter _session_op_remove 5 1 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb3261640 5 client.4197.objecter 0 in flight 2024-09-11T05:07:48.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 10 librados: Objecter returned from create r=0 2024-09-11T05:07:48.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 10 client.4197.objecter _op_submit op 0x55d3f2f545f0 2024-09-11T05:07:48.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 20 client.4197.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:07:48.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 20 client.4197.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-11T05:07:48.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 10 client.4197.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-11T05:07:48.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 20 client.4197.objecter _get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:48.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 10 client.4197.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-11T05:07:48.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 20 client.4197.objecter get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:48.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 15 client.4197.objecter _session_op_assign 5 2 2024-09-11T05:07:48.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 15 client.4197.objecter _send_op 2 to 2.2 on osd.5 2024-09-11T05:07:48.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 --> v2:172.21.15.190:6805/189369151 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55d3f2f54b40 con 0x55d3f2eff0f0 2024-09-11T05:07:48.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:48.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.534+0000 7f4fb34f6100 5 client.4197.objecter 1 in flight 2024-09-11T05:07:48.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 1 -- 172.21.15.138:0/2027105893 <== osd.5 v2:172.21.15.190:6805/189369151 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c008040 con 0x55d3f2eff0f0 2024-09-11T05:07:48.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T05:07:48.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 10 client.4197.objecter in handle_osd_op_reply 2024-09-11T05:07:48.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 7 client.4197.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-11T05:07:48.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 10 client.4197.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T05:07:48.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 10 client.4197.objecter op 0 rval 0 len 0 2024-09-11T05:07:48.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 15 client.4197.objecter handle_osd_op_reply completed tid 2 2024-09-11T05:07:48.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 15 client.4197.objecter _finish_op 2 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 15 client.4197.objecter _session_op_remove 5 2 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 5 client.4197.objecter 0 in flight 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 10 client.4197.objecter _op_submit op 0x55d3f2f53780 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 20 client.4197.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 20 client.4197.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 10 client.4197.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 20 client.4197.objecter _get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 10 client.4197.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 20 client.4197.objecter get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:48.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 15 client.4197.objecter _session_op_assign 5 3 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 15 client.4197.objecter _send_op 3 to 2.2 on osd.5 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 --> v2:172.21.15.190:6805/189369151 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55d3f2f53c30 con 0x55d3f2eff0f0 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb34f6100 5 client.4197.objecter 1 in flight 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 1 -- 172.21.15.138:0/2027105893 <== osd.5 v2:172.21.15.190:6805/189369151 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c008040 con 0x55d3f2eff0f0 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 10 client.4197.objecter in handle_osd_op_reply 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 7 client.4197.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 10 client.4197.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 10 client.4197.objecter op 0 rval 0 len 0 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 15 client.4197.objecter handle_osd_op_reply completed tid 3 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 15 client.4197.objecter _finish_op 3 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:48.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 15 client.4197.objecter _session_op_remove 5 3 2024-09-11T05:07:48.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fb3261640 5 client.4197.objecter 0 in flight 2024-09-11T05:07:48.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fab7fe640 10 client.4197.objecter allocate_selfmanaged_snap; pool: 2 2024-09-11T05:07:48.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fab7fe640 10 client.4197.objecter pool_op_submit 4 2024-09-11T05:07:48.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fab7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:48.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:48.538+0000 7f4fab7fe640 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f4f880030e0 con 0x55d3f2e4c4e0 2024-09-11T05:07:49.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (secure 0 0 0) 0x7f4fa4014c90 con 0x55d3f2e4c4e0 2024-09-11T05:07:49.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 pool_op_reply(tid 4 (0) Success v18) 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 10 client.4197.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v18) 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 10 client.4197.objecter have request 4 at 0x7f4f88003070 Op: create unmanaged snap 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 20 client.4197.objecter waiting for client to reach epoch 18 before calling back 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 10 monclient: _renew_subs 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=18}) -- 0x7f4f98040ef0 con 0x55d3f2e4c4e0 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 10 client.4197.objecter done 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 10 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (secure 0 0 0) 0x7f4fa40b22f0 con 0x55d3f2e4c4e0 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_map(18..18 src has 1..18) 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 3 client.4197.objecter handle_osd_map got epochs [18,18] > 17 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 3 client.4197.objecter handle_osd_map decoding incremental epoch 18 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fa9ffb640 20 client.4197.objecter dump_active .. 0 homeless 2024-09-11T05:07:49.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 20 librados: queue_aio_write 0x55d3f2f4fb80 completion 0x55d3f2ee5ec0 write_seq 1 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 10 client.4197.objecter _op_submit op 0x7f4f8c006390 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 20 client.4197.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 20 client.4197.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 10 client.4197.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 20 client.4197.objecter _get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 10 client.4197.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 20 client.4197.objecter get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 15 client.4197.objecter _session_op_assign 5 5 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 15 client.4197.objecter _send_op 5 to 2.2 on osd.5 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 1 -- 172.21.15.138:0/2027105893 --> v2:172.21.15.190:6805/189369151 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f4f8c006950 con 0x55d3f2eff0f0 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:49.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.518+0000 7f4fabfff640 5 client.4197.objecter 1 in flight 2024-09-11T05:07:49.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 1 -- 172.21.15.138:0/2027105893 <== osd.5 v2:172.21.15.190:6805/189369151 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f4f9c008040 con 0x55d3f2eff0f0 2024-09-11T05:07:49.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) 2024-09-11T05:07:49.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 10 client.4197.objecter in handle_osd_op_reply 2024-09-11T05:07:49.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 7 client.4197.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-11T05:07:49.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 10 client.4197.objecter op 0 rval 0 len 0 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 10 client.4197.objecter op 1 rval 0 len 0 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 15 client.4197.objecter handle_osd_op_reply completed tid 5 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 15 client.4197.objecter _finish_op 5 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 15 client.4197.objecter _session_op_remove 5 5 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 5 client.4197.objecter 0 in flight 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fb3261640 20 librados: complete_aio_write 0x55d3f2ee5ec0 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fab7fe640 10 client.4197.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fab7fe640 10 client.4197.objecter pool_op_submit 6 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fab7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:49.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:49.526+0000 7f4fab7fe640 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7f4f88004530 con 0x55d3f2e4c4e0 2024-09-11T05:07:50.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (secure 0 0 0) 0x7f4fa40bce10 con 0x55d3f2e4c4e0 2024-09-11T05:07:50.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 pool_op_reply(tid 6 (0) Success v19) 2024-09-11T05:07:50.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 10 client.4197.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v19) 2024-09-11T05:07:50.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 10 client.4197.objecter have request 6 at 0x7f4f88003070 Op: delete unmanaged snap 2024-09-11T05:07:50.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 20 client.4197.objecter waiting for client to reach epoch 19 before calling back 2024-09-11T05:07:50.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:50.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 10 monclient: _renew_subs 2024-09-11T05:07:50.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:50.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=19}) -- 0x7f4f98040ef0 con 0x55d3f2e4c4e0 2024-09-11T05:07:50.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 10 client.4197.objecter done 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 1 -- 172.21.15.138:0/2027105893 <== mon.0 v2:172.21.15.138:3300/0 12 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (secure 0 0 0) 0x7f4fa40b24f0 con 0x55d3f2e4c4e0 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_map(19..19 src has 1..19) 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 3 client.4197.objecter handle_osd_map got epochs [19,19] > 18 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 3 client.4197.objecter handle_osd_map decoding incremental epoch 19 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fa9ffb640 20 client.4197.objecter dump_active .. 0 homeless 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 20 librados: queue_aio_write 0x55d3f2f4fb80 completion 0x7f4f88001d70 write_seq 2 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 10 client.4197.objecter _op_submit op 0x7f4f8c006390 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 20 client.4197.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 20 client.4197.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 10 client.4197.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 20 client.4197.objecter _get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:50.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 10 client.4197.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-11T05:07:50.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 20 client.4197.objecter get_session s=0x55d3f2eeba50 osd=5 3 2024-09-11T05:07:50.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 15 client.4197.objecter _session_op_assign 5 7 2024-09-11T05:07:50.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 15 client.4197.objecter _send_op 7 to 2.2 on osd.5 2024-09-11T05:07:50.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 1 -- 172.21.15.138:0/2027105893 --> v2:172.21.15.190:6805/189369151 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f4f8c006d10 con 0x55d3f2eff0f0 2024-09-11T05:07:50.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:50.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.530+0000 7f4fabfff640 5 client.4197.objecter 1 in flight 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 1 -- 172.21.15.138:0/2027105893 <== osd.5 v2:172.21.15.190:6805/189369151 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c008040 con 0x55d3f2eff0f0 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 10 client.4197.objecter ms_dispatch 0x55d3f2d30a30 osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 10 client.4197.objecter in handle_osd_op_reply 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 7 client.4197.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 10 client.4197.objecter op 0 rval 0 len 0 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 15 client.4197.objecter handle_osd_op_reply completed tid 7 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 15 client.4197.objecter _finish_op 7 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 4 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 15 client.4197.objecter _session_op_remove 5 7 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 5 client.4197.objecter 0 in flight 2024-09-11T05:07:50.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 20 librados: complete_aio_write 0x7f4f88001d70 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 10 librados: watch_flush enter 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 10 librados: watch_flush exit 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 10 client.4197.objecter close_session for osd.5 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6805/189369151 conn(0x55d3f2eff0f0 msgr2=0x55d3f2f014e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 --2- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6805/189369151 conn(0x55d3f2eff0f0 0x55d3f2f014e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 20 client.4197.objecter put_session s=0x55d3f2eeba50 osd=5 1 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 20 client.4197.objecter shutdown clearing up homeless session... 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 10 client.4197.objecter successfully canceled tick 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f9803afd0 msgr2=0x7f4f9803d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 --2- 172.21.15.138:0/2027105893 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f9803afd0 0x7f4f9803d490 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x55d3f2eee110 tx=0x7f4fa0005b90 comp rx=0 tx=0).stop 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 10 monclient: shutdown 2024-09-11T05:07:50.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:50.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 msgr2=0x55d3f2eed6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:50.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 --2- 172.21.15.138:0/2027105893 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x55d3f2e4c4e0 0x55d3f2eed6e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4fa40ae040 tx=0x7f4fa4017d20 comp rx=0 tx=0).stop 2024-09-11T05:07:50.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb3261640 1 -- 172.21.15.138:0/2027105893 reap_dead start 2024-09-11T05:07:50.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 shutdown_connections 2024-09-11T05:07:50.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 >> 172.21.15.138:0/2027105893 conn(0x55d3f2e4d5d0 msgr2=0x55d3f2e4f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:50.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 shutdown_connections 2024-09-11T05:07:50.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 -- 172.21.15.138:0/2027105893 wait complete. 2024-09-11T05:07:50.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.538+0000 7f4fb34f6100 1 librados: shutdown 2024-09-11T05:07:50.548 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T05:07:50.549 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T05:07:50.549 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient: get_monmap_and_config 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient: build_initial_monmap 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient: monmap: 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:50.681611+0000 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:50.681611+0000 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:50.680 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:50.681 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:50.681 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:50.681 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:50.681 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:50.681 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 1 Processor -- start 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 1 -- start start 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient: init 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbb20070010 0x7fbb20135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient(hunting): picked mon.noname-c con 0x7fbb20070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbb2009e6e0 0x7fbb201382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient(hunting): picked mon.noname-a con 0x7fbb2009e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:50.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb2009eb10 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb200a8990 con 0x7fbb2009eb10 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fbb200703f0 con 0x7fbb2009e6e0 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbb2009cbf0 con 0x7fbb20070010 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb260fd640 10 monclient(hunting): authenticate will time out at 954.117653s 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbb2009e6e0 0x7fbb201382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbb2009e6e0 0x7fbb201382c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46170/0 (socket says 172.21.15.138:46170) 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 1 -- 172.21.15.138:0/268946999 learned_addr learned my addr 172.21.15.138:0/268946999 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient(hunting): get_auth_request con 0x7fbb2009e6e0 auth_method 0 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:50.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 1 -- 172.21.15.138:0/268946999 >> v2:172.21.15.190:3300/0 conn(0x7fbb20070010 msgr2=0x7fbb20135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 1 --2- 172.21.15.138:0/268946999 >> v2:172.21.15.190:3300/0 conn(0x7fbb20070010 0x7fbb20135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 1 -- 172.21.15.138:0/268946999 >> v1:172.21.15.138:6789/0 conn(0x7fbb2009eb10 legacy=0x7fbb20138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 1 monclient: found mon.noname-a 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:50.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.678+0000 7fbb1effd640 1 -- 172.21.15.138:0/268946999 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2009ced0 con 0x7fbb2009e6e0 2024-09-11T05:07:50.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1effd640 1 -- 172.21.15.138:0/268946999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009e6e0 msgr2=0x7fbb201382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:50.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1effd640 1 --2- 172.21.15.138:0/268946999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009e6e0 0x7fbb201382c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb200665b0 tx=0x7fbb1001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e975f5d2d8f7f18 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 1 -- 172.21.15.138:0/268946999 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb100a5720 con 0x7fbb2009e6e0 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: dump: 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:50.687 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 1 -- 172.21.15.138:0/268946999 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb100a58c0 con 0x7fbb2009e6e0 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 5 monclient: authenticate success, global_id 4200 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 4 set_mon_vals no callback set 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 1 -- 172.21.15.138:0/268946999 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb100a5c90 con 0x7fbb2009e6e0 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:50.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb1e7fc640 10 monclient: dump: 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:50.689 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:50.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 10 monclient: shutdown 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 1 -- 172.21.15.138:0/268946999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009e6e0 msgr2=0x7fbb201382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 1 --2- 172.21.15.138:0/268946999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009e6e0 0x7fbb201382c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb200665b0 tx=0x7fbb1001c880 comp rx=0 tx=0).stop 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 1 -- 172.21.15.138:0/268946999 shutdown_connections 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 1 --2- 172.21.15.138:0/268946999 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009e6e0 0x7fbb201382c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 1 --2- 172.21.15.138:0/268946999 >> v2:172.21.15.190:3300/0 conn(0x7fbb20070010 0x7fbb20135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 1 -- 172.21.15.138:0/268946999 >> 172.21.15.138:0/268946999 conn(0x7fbb200ab830 msgr2=0x7fbb200a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 1 -- 172.21.15.138:0/268946999 shutdown_connections 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.682+0000 7fbb260fd640 1 -- 172.21.15.138:0/268946999 wait complete. 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient: build_initial_monmap 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient: monmap: 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:50.695 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 librados: starting msgr at 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 librados: starting objecter 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 Processor -- start 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 -- start start 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 librados: setting wanted keys 2024-09-11T05:07:50.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 librados: calling monclient init 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient: init 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb20070010 0x7fbb201d0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient(hunting): picked mon.noname-c con 0x7fbb20070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb2009e6e0 0x7fbb2013f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient(hunting): picked mon.noname-b con 0x7fbb2009e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009eb10 0x7fbb2013fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient(hunting): picked mon.noname-a con 0x7fbb2009eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20142e80 con 0x7fbb2009eb10 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:50.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20139120 con 0x7fbb2009e6e0 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbb20143000 con 0x7fbb20070010 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb260fd640 10 monclient(hunting): authenticate will time out at 954.125443s 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009eb10 0x7fbb2013fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb20070010 0x7fbb201d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009eb10 0x7fbb2013fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46174/0 (socket says 172.21.15.138:46174) 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 -- 172.21.15.138:0/642420489 learned_addr learned my addr 172.21.15.138:0/642420489 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient(hunting): get_auth_request con 0x7fbb2009eb10 auth_method 0 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:50.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb20070010 0x7fbb201d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58644/0 (socket says 172.21.15.138:58644) 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 10 monclient(hunting): get_auth_request con 0x7fbb20070010 auth_method 0 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1effd640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb2009e6e0 0x7fbb2013f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1effd640 10 monclient(hunting): get_auth_request con 0x7fbb2009e6e0 auth_method 0 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:50.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient(hunting): handle_auth_done global_id 4203 payload 824 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 -- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb20070010 msgr2=0x7fbb201d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb20070010 0x7fbb201d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 -- 172.21.15.138:0/642420489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb2009e6e0 msgr2=0x7fbb2013f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb2009e6e0 0x7fbb2013f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 monclient: found mon.noname-a 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1ffff640 1 -- 172.21.15.138:0/642420489 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2013a980 con 0x7fbb2009eb10 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.686+0000 7fbb1f7fe640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb20070010 0x7fbb201d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:50.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb1ffff640 1 -- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009eb10 msgr2=0x7fbb2013fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb1ffff640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009eb10 0x7fbb2013fbc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800c3c0 tx=0x7fbb1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 client.?.objecter ms_handle_connect 0x7fbb2009eb10 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 1 -- 172.21.15.138:0/642420489 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb180119a0 con 0x7fbb2009eb10 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: dump: 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:50.701 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: _finish_auth 0 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 1 -- 172.21.15.138:0/642420489 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb18017070 con 0x7fbb2009eb10 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 1 -- 172.21.15.138:0/642420489 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18011d80 con 0x7fbb2009eb10 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 monclient: dump: 2024-09-11T05:07:50.702 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb1dffb640 4 set_mon_vals no callback set 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 5 monclient: authenticate success, global_id 4203 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 10 monclient: _renew_subs 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:50.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb200a2330 con 0x7fbb2009eb10 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 10 client.4203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 10 monclient: _renew_subs 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2013b220 con 0x7fbb2009eb10 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 1 librados: init done 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb20135450 con 0x7fbb2009eb10 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 1 -- 172.21.15.138:0/642420489 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fbb1801c460 con 0x7fbb2009eb10 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 1 --2- 172.21.15.138:0/642420489 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb0c03cef0 0x7fbb0c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 1 -- 172.21.15.138:0/642420489 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fbb1804eff0 con 0x7fbb2009eb10 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 10 client.4203.objecter ms_dispatch 0x7fbb20000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:50.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 3 client.4203.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:50.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 3 client.4203.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:50.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb07fff640 20 client.4203.objecter dump_active .. 0 homeless 2024-09-11T05:07:50.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb1f7fe640 1 --2- 172.21.15.138:0/642420489 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb0c03cef0 0x7fbb0c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:50.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.690+0000 7fbb1f7fe640 10 monclient: get_auth_request con 0x7fbb0c03cef0 auth_method 0 2024-09-11T05:07:50.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.694+0000 7fbb1f7fe640 1 --2- 172.21.15.138:0/642420489 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb0c03cef0 0x7fbb0c03f3b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009d60 tx=0x7fbb14009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:50.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.694+0000 7fbb07fff640 10 client.4203.objecter ms_handle_connect 0x7fbb0c03cef0 2024-09-11T05:07:50.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.694+0000 7fbb07fff640 1 -- 172.21.15.138:0/642420489 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbb2013a980 con 0x7fbb2009eb10 2024-09-11T05:07:50.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.694+0000 7fbb07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:50.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.694+0000 7fbb07fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:50.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.862+0000 7fbb260fd640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T05:07:50.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.862+0000 7fbb260fd640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T05:07:50.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.862+0000 7fbb260fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:50.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.862+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fbb200aa4a0 con 0x7fbb2009eb10 2024-09-11T05:07:50.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb07fff640 1 -- 172.21.15.138:0/642420489 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v3) ==== 86+0+159 (secure 0 0 0) 0x7fbb1801ac60 con 0x7fbb2009eb10 2024-09-11T05:07:50.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T05:07:50.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb07fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:50.868 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 10 librados: watch_flush enter 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 10 librados: watch_flush exit 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 20 client.4203.objecter shutdown clearing up homeless session... 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 10 client.4203.objecter successfully canceled tick 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb0c03cef0 msgr2=0x7fbb0c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 --2- 172.21.15.138:0/642420489 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb0c03cef0 0x7fbb0c03f3b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009d60 tx=0x7fbb14009110 comp rx=0 tx=0).stop 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 10 monclient: shutdown 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:50.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009eb10 msgr2=0x7fbb2013fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009eb10 0x7fbb2013fbc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800c3c0 tx=0x7fbb1800c880 comp rx=0 tx=0).stop 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 shutdown_connections 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 --2- 172.21.15.138:0/642420489 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb0c03cef0 0x7fbb0c03f3b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb2009eb10 0x7fbb2013fbc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb2009e6e0 0x7fbb2013f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 --2- 172.21.15.138:0/642420489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb20070010 0x7fbb201d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 >> 172.21.15.138:0/642420489 conn(0x7fbb200ab830 msgr2=0x7fbb200a9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 shutdown_connections 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 -- 172.21.15.138:0/642420489 wait complete. 2024-09-11T05:07:50.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:50.866+0000 7fbb260fd640 1 librados: shutdown 2024-09-11T05:07:50.885 INFO:teuthology.orchestra.run.smithi138.stdout:[{"version":3,"timestamp":"2024-09-11T05:07:39.375711+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"7990.608542"}]}] 2024-09-11T05:07:50.886 INFO:tasks.ceph_manager:config epoch is 3 2024-09-11T05:07:50.886 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T05:07:50.886 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T05:07:50.886 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T05:07:51.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient: get_monmap_and_config 2024-09-11T05:07:51.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient: monmap: 2024-09-11T05:07:51.007 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.007 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.007 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:51.007946+0000 2024-09-11T05:07:51.007 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:51.007946+0000 2024-09-11T05:07:51.008 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.008 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.008 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:51.008 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:51.008 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:51.008 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:51.009 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:51.009 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 1 Processor -- start 2024-09-11T05:07:51.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 1 -- start start 2024-09-11T05:07:51.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient: init 2024-09-11T05:07:51.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-11T05:07:51.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c1355e0 0x7f665c1359c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient(hunting): picked mon.noname-b con 0x7f665c1355e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:51.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f665c134dd0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:51.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f665c138a60 0x7f665c1406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 10 monclient(hunting): picked mon.noname-a con 0x7f665c138a60 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:51.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.002+0000 7f666352b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f665c074990 con 0x7f665c134dd0 2024-09-11T05:07:51.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f666352b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f666352b640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f665c0a2d00 con 0x7f665c138a60 2024-09-11T05:07:51.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f666352b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f666352b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f665c0a2740 con 0x7f665c1355e0 2024-09-11T05:07:51.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f666352b640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f666352b640 10 monclient(hunting): authenticate will time out at 954.444031s 2024-09-11T05:07:51.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c1355e0 0x7f665c1359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f665c138a60 0x7f665c1406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c1355e0 0x7f665c1359c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58650/0 (socket says 172.21.15.138:58650) 2024-09-11T05:07:51.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 1 -- 172.21.15.138:0/3248281076 learned_addr learned my addr 172.21.15.138:0/3248281076 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:51.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient(hunting): get_auth_request con 0x7f665c1355e0 auth_method 0 2024-09-11T05:07:51.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 10 monclient(hunting): get_auth_request con 0x7f665c138a60 auth_method 0 2024-09-11T05:07:51.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2024-09-11T05:07:51.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 1 -- 172.21.15.138:0/3248281076 >> v2:172.21.15.138:3300/0 conn(0x7f665c138a60 msgr2=0x7f665c1406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 1 --2- 172.21.15.138:0/3248281076 >> v2:172.21.15.138:3300/0 conn(0x7f665c138a60 0x7f665c1406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 1 -- 172.21.15.138:0/3248281076 >> v1:172.21.15.190:6789/0 conn(0x7f665c134dd0 legacy=0x7f665c138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient: _finish_hunting 0 2024-09-11T05:07:51.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 1 monclient: found mon.noname-b 2024-09-11T05:07:51.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:51.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f66612c7640 1 -- 172.21.15.138:0/3248281076 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665c0a2a20 con 0x7f665c1355e0 2024-09-11T05:07:51.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.006+0000 7f6661ac8640 1 --2- 172.21.15.138:0/3248281076 >> v2:172.21.15.138:3300/0 conn(0x7f665c138a60 0x7f665c1406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:51.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f66612c7640 1 -- 172.21.15.138:0/3248281076 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c1355e0 msgr2=0x7f665c1359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:51.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f66612c7640 1 --2- 172.21.15.138:0/3248281076 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c1355e0 0x7f665c1359c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f664c009870 tx=0x7f664c01c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada23af43512a5d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 1 -- 172.21.15.138:0/3248281076 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c0b1070 con 0x7f665c1355e0 2024-09-11T05:07:51.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:51.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: dump: 2024-09-11T05:07:51.039 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.040 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.040 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.040 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.040 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.040 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.041 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.041 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.041 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.041 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: _finish_auth 0 2024-09-11T05:07:51.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:51.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:51.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 1 -- 172.21.15.138:0/3248281076 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f664c0ac410 con 0x7f665c1355e0 2024-09-11T05:07:51.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 5 monclient: authenticate success, global_id 4208 2024-09-11T05:07:51.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:51.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:51.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:51.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 1 -- 172.21.15.138:0/3248281076 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c0ac7e0 con 0x7f665c1355e0 2024-09-11T05:07:51.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 4 set_mon_vals no callback set 2024-09-11T05:07:51.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:51.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6653fff640 10 monclient: dump: 2024-09-11T05:07:51.044 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.044 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.044 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.045 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.045 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.045 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.045 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.045 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.046 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.046 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient: shutdown 2024-09-11T05:07:51.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:51.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- 172.21.15.138:0/3248281076 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c1355e0 msgr2=0x7f665c1359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 --2- 172.21.15.138:0/3248281076 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c1355e0 0x7f665c1359c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f664c009870 tx=0x7f664c01c510 comp rx=0 tx=0).stop 2024-09-11T05:07:51.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- 172.21.15.138:0/3248281076 shutdown_connections 2024-09-11T05:07:51.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 --2- 172.21.15.138:0/3248281076 >> v2:172.21.15.138:3300/0 conn(0x7f665c138a60 0x7f665c1406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 --2- 172.21.15.138:0/3248281076 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c1355e0 0x7f665c1359c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- 172.21.15.138:0/3248281076 >> 172.21.15.138:0/3248281076 conn(0x7f665c063640 msgr2=0x7f665c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:51.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- 172.21.15.138:0/3248281076 shutdown_connections 2024-09-11T05:07:51.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- 172.21.15.138:0/3248281076 wait complete. 2024-09-11T05:07:51.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient: monmap: 2024-09-11T05:07:51.048 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.048 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:51.048 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:51.049 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:51.049 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.049 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.049 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:51.049 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:51.050 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:51.050 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 librados: starting msgr at 2024-09-11T05:07:51.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 librados: starting objecter 2024-09-11T05:07:51.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 Processor -- start 2024-09-11T05:07:51.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- start start 2024-09-11T05:07:51.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 librados: setting wanted keys 2024-09-11T05:07:51.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 librados: calling monclient init 2024-09-11T05:07:51.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient: init 2024-09-11T05:07:51.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:51.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f665c134dd0 0x7f665c1d0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient(hunting): picked mon.noname-b con 0x7f665c134dd0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:51.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f665c1355e0 0x7f665c1d1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient(hunting): picked mon.noname-a con 0x7f665c1355e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:51.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c138a60 0x7f665c1d1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient(hunting): picked mon.noname-c con 0x7f665c138a60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:51.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f665c1435d0 con 0x7f665c1355e0 2024-09-11T05:07:51.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f665c143450 con 0x7f665c134dd0 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f665c143750 con 0x7f665c138a60 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f666352b640 10 monclient(hunting): authenticate will time out at 954.451138s 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c138a60 0x7f665c1d1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f665c1355e0 0x7f665c1d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c138a60 0x7f665c1d1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58656/0 (socket says 172.21.15.138:58656) 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 -- 172.21.15.138:0/1729887519 learned_addr learned my addr 172.21.15.138:0/1729887519 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f665c1355e0 0x7f665c1d1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46194/0 (socket says 172.21.15.138:46194) 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient(hunting): get_auth_request con 0x7f665c138a60 auth_method 0 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 10 monclient(hunting): get_auth_request con 0x7f665c1355e0 auth_method 0 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f66612c7640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f665c134dd0 0x7f665c1d0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2024-09-11T05:07:51.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 -- 172.21.15.138:0/1729887519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f665c134dd0 msgr2=0x7f665c1d0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f665c134dd0 0x7f665c1d0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 -- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f665c1355e0 msgr2=0x7f665c1d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f665c1355e0 0x7f665c1d1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient: _finish_hunting 0 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 monclient: found mon.noname-c 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6661ac8640 1 -- 172.21.15.138:0/1729887519 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665c13f730 con 0x7f665c138a60 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.010+0000 7f6660ac6640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f665c1355e0 0x7f665c1d1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6661ac8640 1 -- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c138a60 msgr2=0x7f665c1d1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6661ac8640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c138a60 0x7f665c1d1690 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f665400efc0 tx=0x7f665400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 client.?.objecter ms_handle_connect 0x7f665c138a60 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:51.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 1 -- 172.21.15.138:0/1729887519 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6654018030 con 0x7f665c138a60 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: dump: 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 1 -- 172.21.15.138:0/1729887519 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66540039a0 con 0x7f665c138a60 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 5 monclient: authenticate success, global_id 4211 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 10 monclient: _renew_subs 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f665c13d3d0 con 0x7f665c138a60 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 1 -- 172.21.15.138:0/1729887519 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6654003d70 con 0x7f665c138a60 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 monclient: dump: 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.059 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f66537fe640 4 set_mon_vals no callback set 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 10 monclient: _renew_subs 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f665c13b910 con 0x7f665c138a60 2024-09-11T05:07:51.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 1 librados: init done 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 1 -- 172.21.15.138:0/1729887519 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f6654003f50 con 0x7f665c138a60 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665c13f350 con 0x7f665c138a60 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 1 --2- 172.21.15.138:0/1729887519 >> v2:172.21.15.190:6800/1562782189 conn(0x7f664003cea0 0x7f664003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 1 -- 172.21.15.138:0/1729887519 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f6654050060 con 0x7f665c138a60 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 10 client.4211.objecter ms_dispatch 0x7f665c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 3 client.4211.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 3 client.4211.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f6651ffb640 20 client.4211.objecter dump_active .. 0 homeless 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f66612c7640 1 --2- 172.21.15.138:0/1729887519 >> v2:172.21.15.190:6800/1562782189 conn(0x7f664003cea0 0x7f664003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.014+0000 7f66612c7640 10 monclient: get_auth_request con 0x7f664003cea0 auth_method 0 2024-09-11T05:07:51.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.018+0000 7f66612c7640 1 --2- 172.21.15.138:0/1729887519 >> v2:172.21.15.190:6800/1562782189 conn(0x7f664003cea0 0x7f664003f360 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f664c0ac000 tx=0x7f664c0a6040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.018+0000 7f6651ffb640 10 client.4211.objecter ms_handle_connect 0x7f664003cea0 2024-09-11T05:07:51.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.018+0000 7f6651ffb640 1 -- 172.21.15.138:0/1729887519 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f6654022070 con 0x7f665c138a60 2024-09-11T05:07:51.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.018+0000 7f6651ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.018+0000 7f6651ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:51.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.218+0000 7f666352b640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-11T05:07:51.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.218+0000 7f666352b640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-11T05:07:51.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.218+0000 7f666352b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.218+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f665c1359c0 con 0x7f665c138a60 2024-09-11T05:07:51.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f6651ffb640 1 -- 172.21.15.138:0/1729887519 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+93181 (secure 0 0 0) 0x7f6654010040 con 0x7f665c138a60 2024-09-11T05:07:51.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f6651ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-11T05:07:51.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f6651ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:51.225 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:51.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 10 librados: watch_flush enter 2024-09-11T05:07:51.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 10 librados: watch_flush exit 2024-09-11T05:07:51.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 20 client.4211.objecter shutdown clearing up homeless session... 2024-09-11T05:07:51.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 10 client.4211.objecter successfully canceled tick 2024-09-11T05:07:51.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 >> v2:172.21.15.190:6800/1562782189 conn(0x7f664003cea0 msgr2=0x7f664003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 --2- 172.21.15.138:0/1729887519 >> v2:172.21.15.190:6800/1562782189 conn(0x7f664003cea0 0x7f664003f360 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f664c0ac000 tx=0x7f664c0a6040 comp rx=0 tx=0).stop 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 10 monclient: shutdown 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c138a60 msgr2=0x7f665c1d1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c138a60 0x7f665c1d1690 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f665400efc0 tx=0x7f665400c490 comp rx=0 tx=0).stop 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 shutdown_connections 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 --2- 172.21.15.138:0/1729887519 >> v2:172.21.15.190:6800/1562782189 conn(0x7f664003cea0 0x7f664003f360 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f665c138a60 0x7f665c1d1690 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f665c1355e0 0x7f665c1d1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 --2- 172.21.15.138:0/1729887519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f665c134dd0 0x7f665c1d0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 >> 172.21.15.138:0/1729887519 conn(0x7f665c063640 msgr2=0x7f665c132cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:51.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 shutdown_connections 2024-09-11T05:07:51.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 -- 172.21.15.138:0/1729887519 wait complete. 2024-09-11T05:07:51.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.222+0000 7f666352b640 1 librados: shutdown 2024-09-11T05:07:51.241 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":1562782189}]},"active_addr":"172.21.15.190:6800/1562782189","active_change":"2024-09-11T05:07:32.653377+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3354887371}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":2368504511}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":492238477}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":458924806}]}]} 2024-09-11T05:07:51.244 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T05:07:51.244 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T05:07:51.245 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:07:51.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient: get_monmap_and_config 2024-09-11T05:07:51.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient: monmap: 2024-09-11T05:07:51.350 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.350 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.350 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:51.351874+0000 2024-09-11T05:07:51.350 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:51.351874+0000 2024-09-11T05:07:51.350 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.351 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.351 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:51.351 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:51.351 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:51.351 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:51.351 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:51.351 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 1 Processor -- start 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 1 -- start start 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient: init 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e7c0a5b40 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3e7c0a7130 0x7f3e7c13db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient(hunting): picked mon.noname-c con 0x7f3e7c0a7130 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e7c09d840 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7c09d490 con 0x7f3e7c0a5b40 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7c070580 con 0x7f3e7c09d840 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7c09e110 con 0x7f3e7c0a7130 2024-09-11T05:07:51.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.346+0000 7f3e81e70640 10 monclient(hunting): authenticate will time out at 954.787942s 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7b7fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f3e7c0a5b40 0x7f3e7c0a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38374/0 (socket says 172.21.15.138:38374) 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7b7fe640 1 -- 172.21.15.138:0/942664911 learned_addr learned my addr 172.21.15.138:0/942664911 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 1 --2- 172.21.15.138:0/942664911 >> v2:172.21.15.190:3300/0 conn(0x7f3e7c0a7130 0x7f3e7c13db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient(hunting): get_auth_request con 0x7f3e7c0a7130 auth_method 0 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient(hunting): handle_auth_done global_id 4204 payload 274 2024-09-11T05:07:51.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 1 -- 172.21.15.138:0/942664911 >> v1:172.21.15.190:6789/0 conn(0x7f3e7c09d840 legacy=0x7f3e7c140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 1 -- 172.21.15.138:0/942664911 >> v1:172.21.15.138:6789/0 conn(0x7f3e7c0a5b40 legacy=0x7f3e7c0a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:51.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 1 monclient: found mon.noname-c 2024-09-11T05:07:51.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:51.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:07:51.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.350+0000 7f3e7affd640 1 -- 172.21.15.138:0/942664911 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c09cb80 con 0x7f3e7c0a7130 2024-09-11T05:07:51.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7affd640 1 -- 172.21.15.138:0/942664911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a7130 msgr2=0x7f3e7c13db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:51.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7affd640 1 --2- 172.21.15.138:0/942664911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a7130 0x7f3e7c13db40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c0097c0 tx=0x7f3e6c01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f0c22bf2f4c788b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 1 -- 172.21.15.138:0/942664911 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0b0070 con 0x7f3e7c0a7130 2024-09-11T05:07:51.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:51.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: dump: 2024-09-11T05:07:51.357 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:51.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 1 -- 172.21.15.138:0/942664911 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6c0ab600 con 0x7f3e7c0a7130 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 1 -- 172.21.15.138:0/942664911 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0ab9d0 con 0x7f3e7c0a7130 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7a7fc640 10 monclient: dump: 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.359 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.360 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.360 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.360 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 5 monclient: authenticate success, global_id 4204 2024-09-11T05:07:51.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:51.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 4 set_mon_vals no callback set 2024-09-11T05:07:51.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient: shutdown 2024-09-11T05:07:51.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:51.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- 172.21.15.138:0/942664911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a7130 msgr2=0x7f3e7c13db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 --2- 172.21.15.138:0/942664911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a7130 0x7f3e7c13db40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c0097c0 tx=0x7f3e6c01c360 comp rx=0 tx=0).stop 2024-09-11T05:07:51.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- 172.21.15.138:0/942664911 shutdown_connections 2024-09-11T05:07:51.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 --2- 172.21.15.138:0/942664911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a7130 0x7f3e7c13db40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- 172.21.15.138:0/942664911 >> 172.21.15.138:0/942664911 conn(0x7f3e7c0aba10 msgr2=0x7f3e7c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:51.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- 172.21.15.138:0/942664911 shutdown_connections 2024-09-11T05:07:51.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- 172.21.15.138:0/942664911 wait complete. 2024-09-11T05:07:51.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient: monmap: 2024-09-11T05:07:51.373 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.373 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:51.373 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:51.373 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:51.373 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.373 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.374 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:51.374 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:51.374 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:51.374 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 librados: starting msgr at 2024-09-11T05:07:51.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 librados: starting objecter 2024-09-11T05:07:51.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 Processor -- start 2024-09-11T05:07:51.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- start start 2024-09-11T05:07:51.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 librados: setting wanted keys 2024-09-11T05:07:51.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 librados: calling monclient init 2024-09-11T05:07:51.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient: init 2024-09-11T05:07:51.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:07:51.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3e7c09d840 0x7f3e7c1d0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient(hunting): picked mon.noname-c con 0x7f3e7c09d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:51.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a5b40 0x7f3e7c1d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient(hunting): picked mon.noname-b con 0x7f3e7c0a5b40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:51.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e7c0a7130 0x7f3e7c13d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient(hunting): picked mon.noname-a con 0x7f3e7c0a7130 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:51.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c144360 con 0x7f3e7c0a7130 2024-09-11T05:07:51.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c1441e0 con 0x7f3e7c0a5b40 2024-09-11T05:07:51.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c1444e0 con 0x7f3e7c09d840 2024-09-11T05:07:51.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e81e70640 10 monclient(hunting): authenticate will time out at 954.795661s 2024-09-11T05:07:51.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3e7c09d840 0x7f3e7c1d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e7c0a7130 0x7f3e7c13d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3e7c09d840 0x7f3e7c1d0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58670/0 (socket says 172.21.15.138:58670) 2024-09-11T05:07:51.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 1 -- 172.21.15.138:0/943533334 learned_addr learned my addr 172.21.15.138:0/943533334 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:51.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 10 monclient(hunting): get_auth_request con 0x7f3e7c09d840 auth_method 0 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7bfff640 10 monclient(hunting): get_auth_request con 0x7f3e7c0a7130 auth_method 0 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.354+0000 7f3e7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7affd640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a5b40 0x7f3e7c1d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 1 -- 172.21.15.138:0/943533334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a5b40 msgr2=0x7f3e7c1d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a5b40 0x7f3e7c1d0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 1 -- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e7c0a7130 msgr2=0x7f3e7c13d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e7c0a7130 0x7f3e7c13d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 1 monclient: found mon.noname-c 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 1 -- 172.21.15.138:0/943533334 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c121e50 con 0x7f3e7c09d840 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7bfff640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e7c0a7130 0x7f3e7c13d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 1 -- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3e7c09d840 msgr2=0x7f3e7c1d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:51.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e7b7fe640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3e7c09d840 0x7f3e7c1d0670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400c470 tx=0x7f3e6400c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 client.?.objecter ms_handle_connect 0x7f3e7c09d840 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 1 -- 172.21.15.138:0/943533334 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e64007070 con 0x7f3e7c09d840 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: dump: 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.381 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: _finish_auth 0 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 1 -- 172.21.15.138:0/943533334 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e64007210 con 0x7f3e7c09d840 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 5 monclient: authenticate success, global_id 4214 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 10 monclient: _renew_subs 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7c122130 con 0x7f3e7c09d840 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 1 -- 172.21.15.138:0/943533334 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e64007600 con 0x7f3e7c09d840 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 monclient: dump: 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e79ffb640 4 set_mon_vals no callback set 2024-09-11T05:07:51.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 10 monclient: _renew_subs 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7c13f450 con 0x7f3e7c09d840 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 1 librados: init done 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e7c0a5f20 con 0x7f3e7c09d840 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 1 -- 172.21.15.138:0/943533334 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3e6401b400 con 0x7f3e7c09d840 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 1 --2- 172.21.15.138:0/943533334 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3e5c03cea0 0x7f3e5c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 1 -- 172.21.15.138:0/943533334 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f3e6404f030 con 0x7f3e7c09d840 2024-09-11T05:07:51.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 10 client.4214.objecter ms_dispatch 0x7f3e7c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:51.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 3 client.4214.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:51.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 3 client.4214.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:51.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.358+0000 7f3e73fff640 20 client.4214.objecter dump_active .. 0 homeless 2024-09-11T05:07:51.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.362+0000 7f3e7affd640 1 --2- 172.21.15.138:0/943533334 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3e5c03cea0 0x7f3e5c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.362+0000 7f3e7affd640 10 monclient: get_auth_request con 0x7f3e5c03cea0 auth_method 0 2024-09-11T05:07:51.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.362+0000 7f3e7affd640 1 --2- 172.21.15.138:0/943533334 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3e5c03cea0 0x7f3e5c03f360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c0ad000 tx=0x7f3e6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.362+0000 7f3e73fff640 10 client.4214.objecter ms_handle_connect 0x7f3e5c03cea0 2024-09-11T05:07:51.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.366+0000 7f3e73fff640 1 -- 172.21.15.138:0/943533334 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3e6401b900 con 0x7f3e7c09d840 2024-09-11T05:07:51.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.366+0000 7f3e73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.366+0000 7f3e73fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:51.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.534+0000 7f3e81e70640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:51.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.534+0000 7f3e81e70640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:51.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.534+0000 7f3e81e70640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.534+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e7c13ffb0 con 0x7f3e7c09d840 2024-09-11T05:07:51.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.534+0000 7f3e73fff640 1 -- 172.21.15.138:0/943533334 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10945 (secure 0 0 0) 0x7f3e64010d10 con 0x7f3e7c09d840 2024-09-11T05:07:51.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.534+0000 7f3e73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:51.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.534+0000 7f3e73fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:51.540 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":19,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","created":"2024-09-11T05:07:27.386635+0000","modified":"2024-09-11T05:07:50.526058+0000","last_up_change":"2024-09-11T05:07:40.695452+0000","last_in_change":"2024-09-11T05:07:31.585387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:07:40.718263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:07:46.826732+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79670fd2-b5aa-408e-8aa4-18a8fb848972","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":16171034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":16171034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":16171034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":16171034}]},"public_addr":"172.21.15.138:6800/16171034","cluster_addr":"172.21.15.138:6801/16171034","heartbeat_back_addr":"172.21.15.138:6803/16171034","heartbeat_front_addr":"172.21.15.138:6802/16171034","state":["exists","up"]},{"osd":1,"uuid":"e96fe3a6-3058-4c40-aa87-26a89572cdfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":4002974512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4002974512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":4002974512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":4002974512}]},"public_addr":"172.21.15.138:6808/4002974512","cluster_addr":"172.21.15.138:6809/4002974512","heartbeat_back_addr":"172.21.15.138:6811/4002974512","heartbeat_front_addr":"172.21.15.138:6810/4002974512","state":["exists","up"]},{"osd":2,"uuid":"a87dbce5-b78c-4070-90cc-d323fc444f4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":81047329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":81047329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":81047329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":81047329}]},"public_addr":"172.21.15.138:6804/81047329","cluster_addr":"172.21.15.138:6805/81047329","heartbeat_back_addr":"172.21.15.138:6807/81047329","heartbeat_front_addr":"172.21.15.138:6806/81047329","state":["exists","up"]},{"osd":3,"uuid":"e1a7ea3f-62f5-4d1f-b671-be183143eb20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3046884208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3046884208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3046884208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3046884208}]},"public_addr":"172.21.15.190:6809/3046884208","cluster_addr":"172.21.15.190:6810/3046884208","heartbeat_back_addr":"172.21.15.190:6812/3046884208","heartbeat_front_addr":"172.21.15.190:6811/3046884208","state":["exists","up"]},{"osd":4,"uuid":"a1aca44b-3dce-4f17-bc06-69bdca9cba04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":1118550024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1118550024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1118550024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":1118550024}]},"public_addr":"172.21.15.190:6801/1118550024","cluster_addr":"172.21.15.190:6802/1118550024","heartbeat_back_addr":"172.21.15.190:6804/1118550024","heartbeat_front_addr":"172.21.15.190:6803/1118550024","state":["exists","up"]},{"osd":5,"uuid":"87ad0149-d931-4ac6-8787-94f9a9609d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":189369151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":189369151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":189369151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":189369151}]},"public_addr":"172.21.15.190:6805/189369151","cluster_addr":"172.21.15.190:6806/189369151","heartbeat_back_addr":"172.21.15.190:6808/189369151","heartbeat_front_addr":"172.21.15.190:6807/189369151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.795496+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.359385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.802420+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.791133+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.377964+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.428799+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:07:51.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 10 librados: watch_flush enter 2024-09-11T05:07:51.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 10 librados: watch_flush exit 2024-09-11T05:07:51.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 20 client.4214.objecter shutdown clearing up homeless session... 2024-09-11T05:07:51.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 10 client.4214.objecter successfully canceled tick 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3e5c03cea0 msgr2=0x7f3e5c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 --2- 172.21.15.138:0/943533334 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3e5c03cea0 0x7f3e5c03f360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c0ad000 tx=0x7f3e6c008040 comp rx=0 tx=0).stop 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 10 monclient: shutdown 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3e7c09d840 msgr2=0x7f3e7c1d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3e7c09d840 0x7f3e7c1d0670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3e6400c470 tx=0x7f3e6400c930 comp rx=0 tx=0).stop 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 shutdown_connections 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 --2- 172.21.15.138:0/943533334 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3e5c03cea0 0x7f3e5c03f360 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3e7c0a7130 0x7f3e7c13d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e7c0a5b40 0x7f3e7c1d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 --2- 172.21.15.138:0/943533334 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3e7c09d840 0x7f3e7c1d0670 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 >> 172.21.15.138:0/943533334 conn(0x7f3e7c0aba10 msgr2=0x7f3e7c0a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:51.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 shutdown_connections 2024-09-11T05:07:51.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 -- 172.21.15.138:0/943533334 wait complete. 2024-09-11T05:07:51.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.538+0000 7f3e81e70640 1 librados: shutdown 2024-09-11T05:07:51.558 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T05:07:51.559 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:07:51.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.662+0000 7f2526664640 10 monclient: get_monmap_and_config 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.662+0000 7f2526664640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.662+0000 7f2526664640 10 monclient: monmap: 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:51.669297+0000 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:51.669297+0000 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:51.668 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:51.669 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:51.669 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 1 Processor -- start 2024-09-11T05:07:51.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 1 -- start start 2024-09-11T05:07:51.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient: init 2024-09-11T05:07:51.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-11T05:07:51.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f252006e0e0 0x7f2520136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient(hunting): picked mon.noname-c con 0x7f252006e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2520136170 0x7f2520136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient(hunting): picked mon.noname-b con 0x7f2520136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f25200a1920 0x7f25200a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient(hunting): picked mon.noname-a con 0x7f25200a1920 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f252006e4c0 con 0x7f25200a1920 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f25200a3270 con 0x7f252006e0e0 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f25200a0af0 con 0x7f2520136170 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2526664640 10 monclient(hunting): authenticate will time out at 955.105291s 2024-09-11T05:07:51.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f25200a1920 0x7f25200a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f25200a1920 0x7f25200a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46214/0 (socket says 172.21.15.138:46214) 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 -- 172.21.15.138:0/1889796465 learned_addr learned my addr 172.21.15.138:0/1889796465 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient(hunting): get_auth_request con 0x7f25200a1920 auth_method 0 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251f7fe640 1 --2- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2520136170 0x7f2520136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251f7fe640 10 monclient(hunting): get_auth_request con 0x7f2520136170 auth_method 0 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251ffff640 1 --2- 172.21.15.138:0/1889796465 >> v2:172.21.15.190:3300/0 conn(0x7f252006e0e0 0x7f2520136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient(hunting): handle_auth_done global_id 4215 payload 274 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 -- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2520136170 msgr2=0x7f2520136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 --2- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2520136170 0x7f2520136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 -- 172.21.15.138:0/1889796465 >> v2:172.21.15.190:3300/0 conn(0x7f252006e0e0 msgr2=0x7f2520136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 --2- 172.21.15.138:0/1889796465 >> v2:172.21.15.190:3300/0 conn(0x7f252006e0e0 0x7f2520136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient: _finish_hunting 0 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 monclient: found mon.noname-a 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f2524c01640 1 -- 172.21.15.138:0/1889796465 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25200a2f90 con 0x7f25200a1920 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251ffff640 10 monclient: get_auth_request con 0x7f252006e0e0 auth_method 0 2024-09-11T05:07:51.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.666+0000 7f251ffff640 1 --2- 172.21.15.138:0/1889796465 >> v2:172.21.15.190:3300/0 conn(0x7f252006e0e0 0x7f2520136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2524c01640 1 -- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25200a1920 msgr2=0x7f25200a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2524c01640 1 --2- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25200a1920 0x7f25200a1db0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f25200647c0 tx=0x7f251801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98349662682a47a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 1 -- 172.21.15.138:0/1889796465 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25180a5510 con 0x7f25200a1920 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: dump: 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.683 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.684 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.684 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.684 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.684 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.684 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.684 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: _finish_auth 0 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 1 -- 172.21.15.138:0/1889796465 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25180a56b0 con 0x7f25200a1920 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 5 monclient: authenticate success, global_id 4215 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 1 -- 172.21.15.138:0/1889796465 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25180a5a80 con 0x7f25200a1920 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:51.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f251effd640 10 monclient: dump: 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 4 set_mon_vals no callback set 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 10 monclient: shutdown 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:51.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 -- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25200a1920 msgr2=0x7f25200a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 --2- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25200a1920 0x7f25200a1db0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f25200647c0 tx=0x7f251801c8b0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 -- 172.21.15.138:0/1889796465 shutdown_connections 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 --2- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25200a1920 0x7f25200a1db0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 --2- 172.21.15.138:0/1889796465 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2520136170 0x7f2520136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 --2- 172.21.15.138:0/1889796465 >> v2:172.21.15.190:3300/0 conn(0x7f252006e0e0 0x7f2520136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 -- 172.21.15.138:0/1889796465 >> 172.21.15.138:0/1889796465 conn(0x7f2520063640 msgr2=0x7f2520063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 -- 172.21.15.138:0/1889796465 shutdown_connections 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 -- 172.21.15.138:0/1889796465 wait complete. 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 10 monclient: monmap: 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:51.687 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 librados: starting msgr at 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 librados: starting objecter 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 Processor -- start 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 -- start start 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 librados: setting wanted keys 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 1 librados: calling monclient init 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.670+0000 7f2526664640 10 monclient: init 2024-09-11T05:07:51.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f252006e0e0 0x7f25201cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient(hunting): picked mon.noname-b con 0x7f252006e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f25200a1920 0x7f252012fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient(hunting): picked mon.noname-c con 0x7f25200a1920 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2520136170 0x7f25201d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient(hunting): picked mon.noname-a con 0x7f2520136170 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2520140090 con 0x7f2520136170 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f252013ff10 con 0x7f252006e0e0 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2520140210 con 0x7f25200a1920 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient(hunting): authenticate will time out at 955.112227s 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f25200a1920 0x7f252012fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2520136170 0x7f25201d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f25200a1920 0x7f252012fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58690/0 (socket says 172.21.15.138:58690) 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 1 -- 172.21.15.138:0/1090280219 learned_addr learned my addr 172.21.15.138:0/1090280219 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 10 monclient(hunting): get_auth_request con 0x7f25200a1920 auth_method 0 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2520136170 0x7f25201d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46218/0 (socket says 172.21.15.138:46218) 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient(hunting): get_auth_request con 0x7f2520136170 auth_method 0 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251ffff640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f252006e0e0 0x7f25201cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 -- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f25200a1920 msgr2=0x7f252012fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f25200a1920 0x7f252012fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 -- 172.21.15.138:0/1090280219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f252006e0e0 msgr2=0x7f25201cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f252006e0e0 0x7f25201cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient: _finish_hunting 0 2024-09-11T05:07:51.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 monclient: found mon.noname-a 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 -- 172.21.15.138:0/1090280219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2520133910 con 0x7f2520136170 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251f7fe640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f25200a1920 0x7f252012fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 -- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2520136170 msgr2=0x7f25201d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2524c01640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2520136170 0x7f25201d0a40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25180ae000 tx=0x7f2518019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 client.?.objecter ms_handle_connect 0x7f2520136170 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 1 -- 172.21.15.138:0/1090280219 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25180bc030 con 0x7f2520136170 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: dump: 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.692 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: _finish_auth 0 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: _check_auth_tickets 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 1 -- 172.21.15.138:0/1090280219 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25180b8680 con 0x7f2520136170 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 1 -- 172.21.15.138:0/1090280219 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25180a5510 con 0x7f2520136170 2024-09-11T05:07:51.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 5 monclient: authenticate success, global_id 4218 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient: _renew_subs 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252011d710 con 0x7f2520136170 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251e7fc640 4 set_mon_vals no callback set 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f251cff9640 10 monclient: dump: 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.694 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient: _renew_subs 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25201ce360 con 0x7f2520136170 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 librados: init done 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.674+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25200a1db0 con 0x7f2520136170 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251cff9640 1 -- 172.21.15.138:0/1090280219 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f25180b8ad0 con 0x7f2520136170 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251cff9640 1 --2- 172.21.15.138:0/1090280219 >> v2:172.21.15.190:6800/1562782189 conn(0x7f250c03cef0 0x7f250c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251cff9640 1 -- 172.21.15.138:0/1090280219 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f25180e9e90 con 0x7f2520136170 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251cff9640 10 client.4218.objecter ms_dispatch 0x7f2520000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251cff9640 3 client.4218.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251cff9640 3 client.4218.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251cff9640 20 client.4218.objecter dump_active .. 0 homeless 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251ffff640 1 --2- 172.21.15.138:0/1090280219 >> v2:172.21.15.190:6800/1562782189 conn(0x7f250c03cef0 0x7f250c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251ffff640 10 monclient: get_auth_request con 0x7f250c03cef0 auth_method 0 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251ffff640 1 --2- 172.21.15.138:0/1090280219 >> v2:172.21.15.190:6800/1562782189 conn(0x7f250c03cef0 0x7f250c03f3b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f251400b7e0 tx=0x7f2514008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.678+0000 7f251cff9640 10 client.4218.objecter ms_handle_connect 0x7f250c03cef0 2024-09-11T05:07:51.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.682+0000 7f251cff9640 1 -- 172.21.15.138:0/1090280219 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f25180c6070 con 0x7f2520136170 2024-09-11T05:07:51.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.682+0000 7f251cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:51.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.682+0000 7f251cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:51.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.838+0000 7f2526664640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:51.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.838+0000 7f2526664640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:51.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.838+0000 7f2526664640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:51.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.838+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f252006ccd0 con 0x7f2520136170 2024-09-11T05:07:51.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.838+0000 7f251cff9640 1 -- 172.21.15.138:0/1090280219 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10945 (secure 0 0 0) 0x7f2520133910 con 0x7f2520136170 2024-09-11T05:07:51.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.838+0000 7f251cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:51.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.838+0000 7f251cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:51.842 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:51.842 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":19,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","created":"2024-09-11T05:07:27.386635+0000","modified":"2024-09-11T05:07:50.526058+0000","last_up_change":"2024-09-11T05:07:40.695452+0000","last_in_change":"2024-09-11T05:07:31.585387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:07:40.718263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:07:46.826732+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79670fd2-b5aa-408e-8aa4-18a8fb848972","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":16171034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":16171034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":16171034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":16171034}]},"public_addr":"172.21.15.138:6800/16171034","cluster_addr":"172.21.15.138:6801/16171034","heartbeat_back_addr":"172.21.15.138:6803/16171034","heartbeat_front_addr":"172.21.15.138:6802/16171034","state":["exists","up"]},{"osd":1,"uuid":"e96fe3a6-3058-4c40-aa87-26a89572cdfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":4002974512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4002974512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":4002974512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":4002974512}]},"public_addr":"172.21.15.138:6808/4002974512","cluster_addr":"172.21.15.138:6809/4002974512","heartbeat_back_addr":"172.21.15.138:6811/4002974512","heartbeat_front_addr":"172.21.15.138:6810/4002974512","state":["exists","up"]},{"osd":2,"uuid":"a87dbce5-b78c-4070-90cc-d323fc444f4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":81047329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":81047329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":81047329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":81047329}]},"public_addr":"172.21.15.138:6804/81047329","cluster_addr":"172.21.15.138:6805/81047329","heartbeat_back_addr":"172.21.15.138:6807/81047329","heartbeat_front_addr":"172.21.15.138:6806/81047329","state":["exists","up"]},{"osd":3,"uuid":"e1a7ea3f-62f5-4d1f-b671-be183143eb20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3046884208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3046884208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3046884208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3046884208}]},"public_addr":"172.21.15.190:6809/3046884208","cluster_addr":"172.21.15.190:6810/3046884208","heartbeat_back_addr":"172.21.15.190:6812/3046884208","heartbeat_front_addr":"172.21.15.190:6811/3046884208","state":["exists","up"]},{"osd":4,"uuid":"a1aca44b-3dce-4f17-bc06-69bdca9cba04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":1118550024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1118550024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1118550024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":1118550024}]},"public_addr":"172.21.15.190:6801/1118550024","cluster_addr":"172.21.15.190:6802/1118550024","heartbeat_back_addr":"172.21.15.190:6804/1118550024","heartbeat_front_addr":"172.21.15.190:6803/1118550024","state":["exists","up"]},{"osd":5,"uuid":"87ad0149-d931-4ac6-8787-94f9a9609d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":189369151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":189369151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":189369151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":189369151}]},"public_addr":"172.21.15.190:6805/189369151","cluster_addr":"172.21.15.190:6806/189369151","heartbeat_back_addr":"172.21.15.190:6808/189369151","heartbeat_front_addr":"172.21.15.190:6807/189369151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.795496+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.359385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.802420+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.791133+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.377964+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.428799+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:07:51.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 10 librados: watch_flush enter 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 10 librados: watch_flush exit 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 20 client.4218.objecter shutdown clearing up homeless session... 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 10 client.4218.objecter successfully canceled tick 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 >> v2:172.21.15.190:6800/1562782189 conn(0x7f250c03cef0 msgr2=0x7f250c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 --2- 172.21.15.138:0/1090280219 >> v2:172.21.15.190:6800/1562782189 conn(0x7f250c03cef0 0x7f250c03f3b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f251400b7e0 tx=0x7f2514008040 comp rx=0 tx=0).stop 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 10 monclient: shutdown 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2520136170 msgr2=0x7f25201d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2520136170 0x7f25201d0a40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25180ae000 tx=0x7f2518019650 comp rx=0 tx=0).stop 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 shutdown_connections 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 --2- 172.21.15.138:0/1090280219 >> v2:172.21.15.190:6800/1562782189 conn(0x7f250c03cef0 0x7f250c03f3b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2520136170 0x7f25201d0a40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f25200a1920 0x7f252012fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 --2- 172.21.15.138:0/1090280219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f252006e0e0 0x7f25201cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 >> 172.21.15.138:0/1090280219 conn(0x7f2520063640 msgr2=0x7f25200a5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:51.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 shutdown_connections 2024-09-11T05:07:51.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 -- 172.21.15.138:0/1090280219 wait complete. 2024-09-11T05:07:51.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.842+0000 7f2526664640 1 librados: shutdown 2024-09-11T05:07:51.860 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T05:07:51.860 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T05:07:51.861 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T05:07:51.861 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T05:07:51.862 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T05:07:51.862 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T05:07:51.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.970+0000 7fadb193e640 10 monclient: get_monmap_and_config 2024-09-11T05:07:51.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.970+0000 7fadb193e640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.970+0000 7fadb193e640 10 monclient: monmap: 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:51.976866+0000 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:51.976866+0000 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:51.976 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:51.977 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:51.977 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 1 Processor -- start 2024-09-11T05:07:51.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 1 -- start start 2024-09-11T05:07:51.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient: init 2024-09-11T05:07:51.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-11T05:07:51.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fadac070010 0x7fadac135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient(hunting): picked mon.noname-a con 0x7fadac070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:51.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fadac09e6e0 0x7fadac1382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient(hunting): picked mon.noname-c con 0x7fadac09e6e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:51.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09eb10 0x7fadac138800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient(hunting): picked mon.noname-b con 0x7fadac09eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fadac0703f0 con 0x7fadac070010 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fadac0a8990 con 0x7fadac09e6e0 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fadac09cbf0 con 0x7fadac09eb10 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadb193e640 10 monclient(hunting): authenticate will time out at 955.412921s 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fadac070010 0x7fadac135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fadac070010 0x7fadac135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46234/0 (socket says 172.21.15.138:46234) 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 1 -- 172.21.15.138:0/2365769277 learned_addr learned my addr 172.21.15.138:0/2365769277 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 10 monclient(hunting): get_auth_request con 0x7fadac070010 auth_method 0 2024-09-11T05:07:51.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 1 --2- 172.21.15.138:0/2365769277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09eb10 0x7fadac138800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient(hunting): get_auth_request con 0x7fadac09eb10 auth_method 0 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaa7fc640 1 --2- 172.21.15.138:0/2365769277 >> v2:172.21.15.190:3300/0 conn(0x7fadac09e6e0 0x7fadac1382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaa7fc640 10 monclient(hunting): get_auth_request con 0x7fadac09e6e0 auth_method 0 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 1 -- 172.21.15.138:0/2365769277 >> v2:172.21.15.190:3300/0 conn(0x7fadac09e6e0 msgr2=0x7fadac1382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 1 --2- 172.21.15.138:0/2365769277 >> v2:172.21.15.190:3300/0 conn(0x7fadac09e6e0 0x7fadac1382c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 1 -- 172.21.15.138:0/2365769277 >> v2:172.21.15.138:3300/0 conn(0x7fadac070010 msgr2=0x7fadac135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 1 --2- 172.21.15.138:0/2365769277 >> v2:172.21.15.138:3300/0 conn(0x7fadac070010 0x7fadac135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:51.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 1 monclient: found mon.noname-b 2024-09-11T05:07:51.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:51.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:51.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.974+0000 7fadab7fe640 1 -- 172.21.15.138:0/2365769277 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadac09ced0 con 0x7fadac09eb10 2024-09-11T05:07:51.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient: get_monmap_and_config 2024-09-11T05:07:51.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient: monmap: 2024-09-11T05:07:51.983 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.983 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.983 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:51.981666+0000 2024-09-11T05:07:51.983 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:51.981666+0000 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 1 Processor -- start 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 1 -- start start 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient: init 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3580a5140 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd3580a4050 0x7fd35813db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient(hunting): picked mon.noname-c con 0x7fd3580a4050 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd3580a4460 0x7fd3581400c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient(hunting): picked mon.noname-a con 0x7fd3580a4460 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd35809dc60 con 0x7fd3580a5140 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fd358074a40 con 0x7fd3580a4460 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd35809cc00 con 0x7fd3580a4050 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35f6c8640 10 monclient(hunting): authenticate will time out at 955.417474s 2024-09-11T05:07:51.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fadab7fe640 1 -- 172.21.15.138:0/2365769277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09eb10 msgr2=0x7fadac138800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:51.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fadab7fe640 1 --2- 172.21.15.138:0/2365769277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09eb10 0x7fadac138800 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fadac06efb0 tx=0x7fada401c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2adeba3cddafbbe8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd3580a4460 0x7fd3581400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd3580a4460 0x7fd3581400c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46236/0 (socket says 172.21.15.138:46236) 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 1 -- 172.21.15.138:0/159059795 learned_addr learned my addr 172.21.15.138:0/159059795 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient(hunting): get_auth_request con 0x7fd3580a4460 auth_method 0 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35cc63640 1 --2- 172.21.15.138:0/159059795 >> v2:172.21.15.190:3300/0 conn(0x7fd3580a4050 0x7fd35813db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:51.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:51.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fada9ffb640 1 -- 172.21.15.138:0/2365769277 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada40b2070 con 0x7fadac09eb10 2024-09-11T05:07:51.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fada9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35cc63640 10 monclient(hunting): get_auth_request con 0x7fd3580a4050 auth_method 0 2024-09-11T05:07:51.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35cc63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:51.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35cc63640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:51.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35cc63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:51.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fada9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:51.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fada9ffb640 10 monclient: dump: 2024-09-11T05:07:51.989 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.989 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.989 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.989 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.989 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fada9ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fada9ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fada9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fadb193e640 5 monclient: authenticate success, global_id 4223 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fadb193e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:51.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fada9ffb640 1 -- 172.21.15.138:0/2365769277 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada40a5750 con 0x7fadac09eb10 2024-09-11T05:07:51.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient(hunting): handle_auth_done global_id 4224 payload 274 2024-09-11T05:07:51.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 1 -- 172.21.15.138:0/159059795 >> v2:172.21.15.190:3300/0 conn(0x7fd3580a4050 msgr2=0x7fd35813db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 1 --2- 172.21.15.138:0/159059795 >> v2:172.21.15.190:3300/0 conn(0x7fd3580a4050 0x7fd35813db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 1 -- 172.21.15.138:0/159059795 >> v1:172.21.15.190:6789/0 conn(0x7fd3580a5140 legacy=0x7fd3580a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:51.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient: _finish_hunting 0 2024-09-11T05:07:51.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 1 monclient: found mon.noname-a 2024-09-11T05:07:51.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:51.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:51.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.978+0000 7fd35dc65640 1 -- 172.21.15.138:0/159059795 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35809cee0 con 0x7fd3580a4460 2024-09-11T05:07:51.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fada9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:51.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fada9ffb640 1 -- 172.21.15.138:0/2365769277 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada40a5b20 con 0x7fadac09eb10 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 4 set_mon_vals no callback set 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fada9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fada9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fada9ffb640 10 monclient: dump: 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:51.994 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient: shutdown 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- 172.21.15.138:0/2365769277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09eb10 msgr2=0x7fadac138800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 --2- 172.21.15.138:0/2365769277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09eb10 0x7fadac138800 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fadac06efb0 tx=0x7fada401c7c0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- 172.21.15.138:0/2365769277 shutdown_connections 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 --2- 172.21.15.138:0/2365769277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09eb10 0x7fadac138800 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 --2- 172.21.15.138:0/2365769277 >> v2:172.21.15.190:3300/0 conn(0x7fadac09e6e0 0x7fadac1382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 --2- 172.21.15.138:0/2365769277 >> v2:172.21.15.138:3300/0 conn(0x7fadac070010 0x7fadac135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- 172.21.15.138:0/2365769277 >> 172.21.15.138:0/2365769277 conn(0x7fadac0ab830 msgr2=0x7fadac0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- 172.21.15.138:0/2365769277 shutdown_connections 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- 172.21.15.138:0/2365769277 wait complete. 2024-09-11T05:07:51.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient: build_initial_monmap 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient: monmap: 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 librados: starting msgr at 2024-09-11T05:07:51.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 librados: starting objecter 2024-09-11T05:07:51.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 Processor -- start 2024-09-11T05:07:51.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- start start 2024-09-11T05:07:51.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 librados: setting wanted keys 2024-09-11T05:07:51.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 librados: calling monclient init 2024-09-11T05:07:51.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient: init 2024-09-11T05:07:51.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35dc65640 1 -- 172.21.15.138:0/159059795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 msgr2=0x7fd3581400c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:51.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35dc65640 1 --2- 172.21.15.138:0/159059795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 0x7fd3581400c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3580a5990 tx=0x7fd35001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948325dc8a4b2b1e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadac070010 0x7fadac1d0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient(hunting): picked mon.noname-b con 0x7fadac070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09e6e0 0x7fadac13e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient(hunting): picked mon.noname-c con 0x7fadac09e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac09eb10 0x7fadac13ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient(hunting): picked mon.noname-a con 0x7fadac09eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fadac142080 con 0x7fadac09eb10 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fadac142200 con 0x7fadac070010 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fadac142380 con 0x7fadac09e6e0 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadb193e640 10 monclient(hunting): authenticate will time out at 955.422569s 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 1 -- 172.21.15.138:0/159059795 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3500a5630 con 0x7fd3580a4460 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: dump: 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:51.999 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 1 -- 172.21.15.138:0/159059795 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3500a57d0 con 0x7fd3580a4460 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 1 -- 172.21.15.138:0/159059795 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3500a5ba0 con 0x7fd3580a4460 2024-09-11T05:07:52.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd34ffff640 10 monclient: dump: 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 5 monclient: authenticate success, global_id 4224 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 4 set_mon_vals no callback set 2024-09-11T05:07:52.002 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09e6e0 0x7fadac13e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.002 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09e6e0 0x7fadac13e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58708/0 (socket says 172.21.15.138:58708) 2024-09-11T05:07:52.002 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 -- 172.21.15.138:0/3167618420 learned_addr learned my addr 172.21.15.138:0/3167618420 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient(hunting): get_auth_request con 0x7fadac09e6e0 auth_method 0 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadab7fe640 1 --2- 172.21.15.138:0/3167618420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac09eb10 0x7fadac13ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadab7fe640 10 monclient(hunting): get_auth_request con 0x7fadac09eb10 auth_method 0 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaaffd640 1 --2- 172.21.15.138:0/3167618420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadac070010 0x7fadac1d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaaffd640 10 monclient(hunting): get_auth_request con 0x7fadac070010 auth_method 0 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 -- 172.21.15.138:0/3167618420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadac070010 msgr2=0x7fadac1d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 --2- 172.21.15.138:0/3167618420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fadac070010 0x7fadac1d0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 -- 172.21.15.138:0/3167618420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac09eb10 msgr2=0x7fadac13ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 --2- 172.21.15.138:0/3167618420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fadac09eb10 0x7fadac13ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fadaa7fc640 1 -- 172.21.15.138:0/3167618420 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadac13b160 con 0x7fadac09e6e0 2024-09-11T05:07:52.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 10 monclient: shutdown 2024-09-11T05:07:52.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 1 -- 172.21.15.138:0/159059795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 msgr2=0x7fd3581400c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 1 --2- 172.21.15.138:0/159059795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 0x7fd3581400c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3580a5990 tx=0x7fd35001c880 comp rx=0 tx=0).stop 2024-09-11T05:07:52.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 1 -- 172.21.15.138:0/159059795 shutdown_connections 2024-09-11T05:07:52.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 1 --2- 172.21.15.138:0/159059795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 0x7fd3581400c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 1 --2- 172.21.15.138:0/159059795 >> v2:172.21.15.190:3300/0 conn(0x7fd3580a4050 0x7fd35813db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 1 -- 172.21.15.138:0/159059795 >> 172.21.15.138:0/159059795 conn(0x7fd3580a8530 msgr2=0x7fd3580a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 1 -- 172.21.15.138:0/159059795 shutdown_connections 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.982+0000 7fd35f6c8640 1 -- 172.21.15.138:0/159059795 wait complete. 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient: monmap: 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.008 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 librados: starting msgr at 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 librados: starting objecter 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 Processor -- start 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 -- start start 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 librados: setting wanted keys 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 librados: calling monclient init 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient: init 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3580a4050 0x7fd35811fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient(hunting): picked mon.noname-b con 0x7fd3580a4050 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 0x7fd358121c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient(hunting): picked mon.noname-a con 0x7fd3580a4460 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd3580a5140 0x7fd358138ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient(hunting): picked mon.noname-c con 0x7fd3580a5140 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd358142e20 con 0x7fd3580a4460 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd35806d6d0 con 0x7fd3580a4050 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd358142fa0 con 0x7fd3580a5140 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35f6c8640 10 monclient(hunting): authenticate will time out at 955.425236s 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd3580a5140 0x7fd358138ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd3580a5140 0x7fd358138ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58718/0 (socket says 172.21.15.138:58718) 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35cc63640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 0x7fd358121c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 -- 172.21.15.138:0/151991219 learned_addr learned my addr 172.21.15.138:0/151991219 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient(hunting): get_auth_request con 0x7fd3580a5140 auth_method 0 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35cc63640 10 monclient(hunting): get_auth_request con 0x7fd3580a4460 auth_method 0 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35cc63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35cc63640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35cc63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35cc63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35cc63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35cc63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35d464640 1 --2- 172.21.15.138:0/151991219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3580a4050 0x7fd35811fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35d464640 10 monclient(hunting): get_auth_request con 0x7fd3580a4050 auth_method 0 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35d464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35d464640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35d464640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 -- 172.21.15.138:0/151991219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3580a4050 msgr2=0x7fd35811fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 --2- 172.21.15.138:0/151991219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3580a4050 0x7fd35811fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 -- 172.21.15.138:0/151991219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 msgr2=0x7fd358121c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 --2- 172.21.15.138:0/151991219 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd3580a4460 0x7fd358121c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fd35dc65640 1 -- 172.21.15.138:0/151991219 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3581210e0 con 0x7fd3580a5140 2024-09-11T05:07:52.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadaa7fc640 1 -- 172.21.15.138:0/3167618420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09e6e0 msgr2=0x7fadac13e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadaa7fc640 1 --2- 172.21.15.138:0/3167618420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09e6e0 0x7fadac13e6b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fada000d830 tx=0x7fada000dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 client.?.objecter ms_handle_connect 0x7fadac09e6e0 2024-09-11T05:07:52.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada000b870 con 0x7fadac09e6e0 2024-09-11T05:07:52.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: dump: 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada0019070 con 0x7fadac09e6e0 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada000bc10 con 0x7fadac09e6e0 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 monclient: dump: 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.016 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 5 monclient: authenticate success, global_id 4226 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 10 monclient: _renew_subs 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadac13f130 con 0x7fadac09e6e0 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 10 monclient: _renew_subs 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fadac139960 con 0x7fadac09e6e0 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 1 librados: init done 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fada97fa640 4 set_mon_vals no callback set 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 10 client.4226.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 15 client.4226.objecter _session_command_op_assign -1 1 2024-09-11T05:07:52.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 20 client.4226.objecter _get_session osd=-1 returning homeless 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fad8c002920 con 0x7fadac09e6e0 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fada0013070 con 0x7fadac09e6e0 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 1 --2- 172.21.15.138:0/3167618420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fad9803cef0 0x7fad9803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fada004fd90 con 0x7fadac09e6e0 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 10 client.4226.objecter ms_dispatch 0x7fadac000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 3 client.4226.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.986+0000 7fad93fff640 3 client.4226.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 10 client.4226.objecter checking command 1 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 1 --2- 172.21.15.138:0/3167618420 >> v2:172.21.15.138:6800/16171034 conn(0x7fad98042a80 0x7fad98044ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 20 client.4226.objecter _get_session s=0x7fad980423d0 osd=0 3 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 20 client.4226.objecter put_session s=0x7fad980423d0 osd=0 3 2024-09-11T05:07:52.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 15 client.4226.objecter _session_command_op_remove -1 1 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 20 client.4226.objecter _get_session s=0x7fad980423d0 osd=0 3 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 20 client.4226.objecter get_session s=0x7fad980423d0 osd=0 3 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 15 client.4226.objecter _session_command_op_assign 0 1 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 20 client.4226.objecter put_session s=0x7fad980423d0 osd=0 4 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 10 client.4226.objecter _send_command 1 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 --> v2:172.21.15.138:6800/16171034 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad98045550 con 0x7fad98042a80 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 20 client.4226.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fada0050070 con 0x7fadac09e6e0 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35dc65640 1 -- 172.21.15.138:0/151991219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd3580a5140 msgr2=0x7fd358138ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35dc65640 1 --2- 172.21.15.138:0/151991219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd3580a5140 0x7fd358138ab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd3500af040 tx=0x7fd350019620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 client.?.objecter ms_handle_connect 0x7fd3580a5140 2024-09-11T05:07:52.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fadab7fe640 1 --2- 172.21.15.138:0/3167618420 >> v2:172.21.15.138:6800/16171034 conn(0x7fad98042a80 0x7fad98044ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fadab7fe640 10 monclient: get_auth_request con 0x7fad98042a80 auth_method 0 2024-09-11T05:07:52.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fadaaffd640 1 --2- 172.21.15.138:0/3167618420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fad9803cef0 0x7fad9803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fadaaffd640 10 monclient: get_auth_request con 0x7fad9803cef0 auth_method 0 2024-09-11T05:07:52.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3500bc020 con 0x7fd3580a5140 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: dump: 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3500b8880 con 0x7fd3580a5140 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3500a5630 con 0x7fd3580a5140 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: dump: 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.023 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34f7fe640 4 set_mon_vals no callback set 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 5 monclient: authenticate success, global_id 4229 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 10 monclient: _renew_subs 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd358123440 con 0x7fd3580a5140 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 10 monclient: _renew_subs 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd358120a30 con 0x7fd3580a5140 2024-09-11T05:07:52.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 1 librados: init done 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 10 client.4229.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 15 client.4229.objecter _session_command_op_assign -1 1 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 20 client.4229.objecter _get_session osd=-1 returning homeless 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd35809f120 con 0x7fd3580a5140 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fd3500b8c70 con 0x7fd3580a5140 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 --2- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd33c03cef0 0x7fd33c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fd3500ea8d0 con 0x7fd3580a5140 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 client.4229.objecter ms_dispatch 0x7fd358000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 3 client.4229.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 3 client.4229.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 client.4229.objecter checking command 1 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 --2- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6805/189369151 conn(0x7fd33c042a80 0x7fd33c044ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 20 client.4229.objecter _get_session s=0x7fd33c0423d0 osd=5 3 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 20 client.4229.objecter put_session s=0x7fd33c0423d0 osd=5 3 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 15 client.4229.objecter _session_command_op_remove -1 1 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 20 client.4229.objecter _get_session s=0x7fd33c0423d0 osd=5 3 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 20 client.4229.objecter get_session s=0x7fd33c0423d0 osd=5 3 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 15 client.4229.objecter _session_command_op_assign 5 1 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 20 client.4229.objecter put_session s=0x7fd33c0423d0 osd=5 4 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 client.4229.objecter _send_command 1 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 --> v2:172.21.15.190:6805/189369151 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3500bae50 con 0x7fd33c042a80 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 20 client.4229.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd3500eabb0 con 0x7fd3580a5140 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd34dffb640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35cc63640 1 --2- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6805/189369151 conn(0x7fd33c042a80 0x7fd33c044ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35cc63640 10 monclient: get_auth_request con 0x7fd33c042a80 auth_method 0 2024-09-11T05:07:52.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fad93fff640 10 client.4226.objecter ms_handle_connect 0x7fad9803cef0 2024-09-11T05:07:52.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fadaaffd640 1 --2- 172.21.15.138:0/3167618420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fad9803cef0 0x7fad9803f3b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fad9c006cd0 tx=0x7fad9c009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.990+0000 7fd35d464640 1 --2- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd33c03cef0 0x7fd33c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fd35d464640 10 monclient: get_auth_request con 0x7fd33c03cef0 auth_method 0 2024-09-11T05:07:52.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fad93fff640 10 client.4226.objecter ms_handle_connect 0x7fad98042a80 2024-09-11T05:07:52.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fadab7fe640 1 --2- 172.21.15.138:0/3167618420 >> v2:172.21.15.138:6800/16171034 conn(0x7fad98042a80 0x7fad98044ec0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 <== osd.0 v2:172.21.15.138:6800/16171034 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fad98045550 con 0x7fad98042a80 2024-09-11T05:07:52.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fad93fff640 10 client.4226.objecter ms_dispatch 0x7fadac000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:07:52.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fad93fff640 10 client.4226.objecter _finish_command 1 = system:0 2024-09-11T05:07:52.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fad93fff640 20 client.4226.objecter put_session s=0x7fad980423d0 osd=0 4 2024-09-11T05:07:52.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fad93fff640 15 client.4226.objecter _session_command_op_remove 0 1 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fd34dffb640 10 client.4229.objecter ms_handle_connect 0x7fd33c03cef0 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.994+0000 7fd35d464640 1 --2- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd33c03cef0 0x7fd33c03f3b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd354004e30 tx=0x7fd354009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.998+0000 7fd34dffb640 10 client.4229.objecter ms_handle_connect 0x7fd33c042a80 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.998+0000 7fd35cc63640 1 --2- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6805/189369151 conn(0x7fd33c042a80 0x7fd33c044ec0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.998+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 <== osd.5 v2:172.21.15.190:6805/189369151 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd3500bae50 con 0x7fd33c042a80 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.998+0000 7fd34dffb640 10 client.4229.objecter ms_dispatch 0x7fd358000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.998+0000 7fd34dffb640 10 client.4229.objecter _finish_command 1 = system:0 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.998+0000 7fd34dffb640 20 client.4229.objecter put_session s=0x7fd33c0423d0 osd=5 4 2024-09-11T05:07:52.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:51.998+0000 7fd34dffb640 15 client.4229.objecter _session_command_op_remove 5 1 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 10 client.4226.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 15 client.4226.objecter _session_command_op_assign -1 2 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 20 client.4226.objecter _get_session s=0x7fad980423d0 osd=0 3 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 20 client.4226.objecter put_session s=0x7fad980423d0 osd=0 3 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 20 client.4226.objecter _get_session s=0x7fad980423d0 osd=0 3 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 15 client.4226.objecter _session_command_op_remove -1 2 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 20 client.4226.objecter get_session s=0x7fad980423d0 osd=0 3 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 15 client.4226.objecter _session_command_op_assign 0 2 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 20 client.4226.objecter put_session s=0x7fad980423d0 osd=0 4 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 10 client.4226.objecter _send_command 2 2024-09-11T05:07:52.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.006+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 --> v2:172.21.15.138:6800/16171034 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fad8c001160 con 0x7fad98042a80 2024-09-11T05:07:52.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fad93fff640 1 -- 172.21.15.138:0/3167618420 <== osd.0 v2:172.21.15.138:6800/16171034 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fad8c001160 con 0x7fad98042a80 2024-09-11T05:07:52.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fad93fff640 10 client.4226.objecter ms_dispatch 0x7fadac000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:07:52.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fad93fff640 10 client.4226.objecter _finish_command 2 = system:0 2024-09-11T05:07:52.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fad93fff640 20 client.4226.objecter put_session s=0x7fad980423d0 osd=0 4 2024-09-11T05:07:52.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fad93fff640 15 client.4226.objecter _session_command_op_remove 0 2 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 10 librados: watch_flush enter 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 10 librados: watch_flush exit 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 10 client.4226.objecter close_session for osd.0 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 >> v2:172.21.15.138:6800/16171034 conn(0x7fad98042a80 msgr2=0x7fad98044ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 --2- 172.21.15.138:0/3167618420 >> v2:172.21.15.138:6800/16171034 conn(0x7fad98042a80 0x7fad98044ec0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 20 client.4226.objecter put_session s=0x7fad980423d0 osd=0 1 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 20 client.4226.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 10 client.4226.objecter successfully canceled tick 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fad9803cef0 msgr2=0x7fad9803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 --2- 172.21.15.138:0/3167618420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fad9803cef0 0x7fad9803f3b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fad9c006cd0 tx=0x7fad9c009240 comp rx=0 tx=0).stop 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 10 monclient: shutdown 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09e6e0 msgr2=0x7fadac13e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 --2- 172.21.15.138:0/3167618420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fadac09e6e0 0x7fadac13e6b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fada000d830 tx=0x7fada000dcf0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadab7fe640 1 -- 172.21.15.138:0/3167618420 reap_dead start 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 shutdown_connections 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 >> 172.21.15.138:0/3167618420 conn(0x7fadac0ab830 msgr2=0x7fadac0a9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 shutdown_connections 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 -- 172.21.15.138:0/3167618420 wait complete. 2024-09-11T05:07:52.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.010+0000 7fadb193e640 1 librados: shutdown 2024-09-11T05:07:52.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 10 client.4229.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T05:07:52.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 15 client.4229.objecter _session_command_op_assign -1 2 2024-09-11T05:07:52.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 20 client.4229.objecter _get_session s=0x7fd33c0423d0 osd=5 3 2024-09-11T05:07:52.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 20 client.4229.objecter put_session s=0x7fd33c0423d0 osd=5 3 2024-09-11T05:07:52.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 20 client.4229.objecter _get_session s=0x7fd33c0423d0 osd=5 3 2024-09-11T05:07:52.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 15 client.4229.objecter _session_command_op_remove -1 2 2024-09-11T05:07:52.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 20 client.4229.objecter get_session s=0x7fd33c0423d0 osd=5 3 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 15 client.4229.objecter _session_command_op_assign 5 2 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 20 client.4229.objecter put_session s=0x7fd33c0423d0 osd=5 4 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 10 client.4229.objecter _send_command 2 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 --> v2:172.21.15.190:6805/189369151 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd35809f120 con 0x7fd33c042a80 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd34dffb640 1 -- 172.21.15.138:0/151991219 <== osd.5 v2:172.21.15.190:6805/189369151 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd35809f120 con 0x7fd33c042a80 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd34dffb640 10 client.4229.objecter ms_dispatch 0x7fd358000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd34dffb640 10 client.4229.objecter _finish_command 2 = system:0 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd34dffb640 20 client.4229.objecter put_session s=0x7fd33c0423d0 osd=5 4 2024-09-11T05:07:52.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd34dffb640 15 client.4229.objecter _session_command_op_remove 5 2 2024-09-11T05:07:52.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 10 librados: watch_flush enter 2024-09-11T05:07:52.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 10 librados: watch_flush exit 2024-09-11T05:07:52.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 10 client.4229.objecter close_session for osd.5 2024-09-11T05:07:52.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6805/189369151 conn(0x7fd33c042a80 msgr2=0x7fd33c044ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 --2- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6805/189369151 conn(0x7fd33c042a80 0x7fd33c044ec0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 20 client.4229.objecter put_session s=0x7fd33c0423d0 osd=5 1 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 20 client.4229.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 10 client.4229.objecter successfully canceled tick 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd33c03cef0 msgr2=0x7fd33c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 --2- 172.21.15.138:0/151991219 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd33c03cef0 0x7fd33c03f3b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd354004e30 tx=0x7fd354009040 comp rx=0 tx=0).stop 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 10 monclient: shutdown 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd3580a5140 msgr2=0x7fd358138ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 --2- 172.21.15.138:0/151991219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd3580a5140 0x7fd358138ab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd3500af040 tx=0x7fd350019620 comp rx=0 tx=0).stop 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35dc65640 1 -- 172.21.15.138:0/151991219 reap_dead start 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 shutdown_connections 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 >> 172.21.15.138:0/151991219 conn(0x7fd3580a8530 msgr2=0x7fd3580a6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 shutdown_connections 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 -- 172.21.15.138:0/151991219 wait complete. 2024-09-11T05:07:52.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.014+0000 7fd35f6c8640 1 librados: shutdown 2024-09-11T05:07:52.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient: monmap: 2024-09-11T05:07:52.039 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.039 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.034478+0000 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.034478+0000 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 1 Processor -- start 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 1 -- start start 2024-09-11T05:07:52.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient: init 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f988edd3640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f988edd3640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f988edd3640 10 monclient: monmap: 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.035556+0000 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.035556+0000 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.041 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.042 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.042 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.042 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.042 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.042 INFO:teuthology.orchestra.run.smithi138.stdout:47244640261 2024-09-11T05:07:52.043 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0fc8070010 0x7f0fc8135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc8070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0fc809e6e0 0x7f0fc809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc809e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fc81382c0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc80a8990 con 0x7f0fc81382c0 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc80703f0 con 0x7f0fc8070010 2024-09-11T05:07:52.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc809cbf0 con 0x7f0fc809e6e0 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fcd831640 10 monclient(hunting): authenticate will time out at 955.470706s 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0fc8070010 0x7f0fc8135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc67fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0fc809e6e0 0x7f0fc809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0fc8070010 0x7f0fc8135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46254/0 (socket says 172.21.15.138:46254) 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 1 -- 172.21.15.138:0/985114716 learned_addr learned my addr 172.21.15.138:0/985114716 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient(hunting): get_auth_request con 0x7f0fc8070010 auth_method 0 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc67fc640 10 monclient(hunting): get_auth_request con 0x7f0fc809e6e0 auth_method 0 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.046 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f988edd3640 1 Processor -- start 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 1 -- 172.21.15.138:0/985114716 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0fc809e6e0 msgr2=0x7f0fc809eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 1 --2- 172.21.15.138:0/985114716 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0fc809e6e0 0x7f0fc809eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 1 -- 172.21.15.138:0/985114716 >> v1:172.21.15.190:6789/0 conn(0x7f0fc81382c0 legacy=0x7f0fc8138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 1 monclient: found mon.noname-a 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f0fc6ffd640 1 -- 172.21.15.138:0/985114716 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc809ced0 con 0x7f0fc8070010 2024-09-11T05:07:52.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f988edd3640 1 -- start start 2024-09-11T05:07:52.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.030+0000 7f988edd3640 10 monclient: init 2024-09-11T05:07:52.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-11T05:07:52.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880026410 0x7f9880023df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient(hunting): picked mon.noname-b con 0x7f9880026410 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f9880024330 0x7f9880006650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient(hunting): picked mon.noname-a con 0x7f9880024330 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9880006b90 0x7f9880007000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient(hunting): picked mon.noname-c con 0x7f9880006b90 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f98800b9ca0 con 0x7f9880024330 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9880020920 con 0x7f9880006b90 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f9880022020 con 0x7f9880026410 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988edd3640 10 monclient(hunting): authenticate will time out at 955.472359s 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880026410 0x7f9880023df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880026410 0x7f9880023df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58736/0 (socket says 172.21.15.138:58736) 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 -- 172.21.15.138:0/2921608675 learned_addr learned my addr 172.21.15.138:0/2921608675 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient(hunting): get_auth_request con 0x7f9880026410 auth_method 0 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988e5d2640 1 --2- 172.21.15.138:0/2921608675 >> v2:172.21.15.190:3300/0 conn(0x7f9880006b90 0x7f9880007000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988e5d2640 10 monclient(hunting): get_auth_request con 0x7f9880006b90 auth_method 0 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988e5d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988e5d2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988e5d2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 -- 172.21.15.138:0/2921608675 >> v2:172.21.15.190:3300/0 conn(0x7f9880006b90 msgr2=0x7f9880007000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988d5d0640 1 --2- 172.21.15.138:0/2921608675 >> v2:172.21.15.138:3300/0 conn(0x7f9880024330 0x7f9880006650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 --2- 172.21.15.138:0/2921608675 >> v2:172.21.15.190:3300/0 conn(0x7f9880006b90 0x7f9880007000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 -- 172.21.15.138:0/2921608675 >> v2:172.21.15.138:3300/0 conn(0x7f9880024330 msgr2=0x7f9880006650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 --2- 172.21.15.138:0/2921608675 >> v2:172.21.15.138:3300/0 conn(0x7f9880024330 0x7f9880006650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 monclient: found mon.noname-b 2024-09-11T05:07:52.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988ddd1640 1 -- 172.21.15.138:0/2921608675 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9880022300 con 0x7f9880026410 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient: monmap: 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.038431+0000 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.038431+0000 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.054 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.055 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.055 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.055 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.055 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988e5d2640 1 --2- 172.21.15.138:0/2921608675 >> v2:172.21.15.190:3300/0 conn(0x7f9880006b90 0x7f9880007000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:07:52.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988d5d0640 10 monclient: get_auth_request con 0x7f9880024330 auth_method 0 2024-09-11T05:07:52.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f988d5d0640 1 --2- 172.21.15.138:0/2921608675 >> v2:172.21.15.138:3300/0 conn(0x7f9880024330 0x7f9880006650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient: monmap: 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.038995+0000 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.038995+0000 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.056 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.057 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.057 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.057 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 1 Processor -- start 2024-09-11T05:07:52.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 1 -- start start 2024-09-11T05:07:52.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient: init 2024-09-11T05:07:52.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 1 Processor -- start 2024-09-11T05:07:52.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 1 -- start start 2024-09-11T05:07:52.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient: init 2024-09-11T05:07:52.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1a880a37d0 0x7f1a880a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient(hunting): picked mon.noname-a con 0x7f1a880a37d0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1a8809d8e0 0x7f1a880a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient(hunting): picked mon.noname-c con 0x7f1a8809d8e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a8806d470 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a8809d530 con 0x7f1a8806d470 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f1a88075190 con 0x7f1a880a37d0 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1a8806cbf0 con 0x7f1a8809d8e0 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a8d59e640 10 monclient(hunting): authenticate will time out at 955.474555s 2024-09-11T05:07:52.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-11T05:07:52.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f26380a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient(hunting): picked mon.noname-b con 0x7f26380a3940 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1a880a37d0 0x7f1a880a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a877fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f1a8806d470 0x7f1a881385b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38390/0 (socket says 172.21.15.138:38390) 2024-09-11T05:07:52.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f26380a0820 0x7f26380a4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient(hunting): picked mon.noname-a con 0x7f26380a0820 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:52.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2638136700 0x7f2638136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient(hunting): picked mon.noname-c con 0x7f2638136700 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:52.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f26380a47a0 con 0x7f26380a0820 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f26380a2110 con 0x7f2638136700 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f263809fe40 con 0x7f26380a3940 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f263deef640 10 monclient(hunting): authenticate will time out at 955.474865s 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a877fe640 1 -- 172.21.15.138:0/1741132804 learned_addr learned my addr 172.21.15.138:0/1741132804 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient(hunting): get_auth_request con 0x7f1a880a37d0 auth_method 0 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.063 INFO:teuthology.orchestra.run.smithi138.stdout:47244640260 2024-09-11T05:07:52.063 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T05:07:52.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f26380a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f26380a3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58746/0 (socket says 172.21.15.138:58746) 2024-09-11T05:07:52.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 -- 172.21.15.138:0/3821831476 learned_addr learned my addr 172.21.15.138:0/3821831476 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient(hunting): get_auth_request con 0x7f26380a3940 auth_method 0 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a867fc640 1 --2- 172.21.15.138:0/1741132804 >> v2:172.21.15.190:3300/0 conn(0x7f1a8809d8e0 0x7f1a880a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a867fc640 10 monclient(hunting): get_auth_request con 0x7f1a8809d8e0 auth_method 0 2024-09-11T05:07:52.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a867fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-09-11T05:07:52.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 1 -- 172.21.15.138:0/1741132804 >> v2:172.21.15.190:3300/0 conn(0x7f1a8809d8e0 msgr2=0x7f1a880a40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 1 --2- 172.21.15.138:0/1741132804 >> v2:172.21.15.190:3300/0 conn(0x7f1a8809d8e0 0x7f1a880a40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 1 -- 172.21.15.138:0/1741132804 >> v1:172.21.15.138:6789/0 conn(0x7f1a8806d470 legacy=0x7f1a881385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f2637fff640 1 --2- 172.21.15.138:0/3821831476 >> v2:172.21.15.190:3300/0 conn(0x7f2638136700 0x7f2638136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 1 monclient: found mon.noname-a 2024-09-11T05:07:52.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f1a86ffd640 1 -- 172.21.15.138:0/1741132804 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8809cc20 con 0x7f1a880a37d0 2024-09-11T05:07:52.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f2637fff640 10 monclient(hunting): get_auth_request con 0x7f2638136700 auth_method 0 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f2637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f2637fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f2637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 -- 172.21.15.138:0/3821831476 >> v2:172.21.15.190:3300/0 conn(0x7f2638136700 msgr2=0x7f2638136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 --2- 172.21.15.138:0/3821831476 >> v2:172.21.15.190:3300/0 conn(0x7f2638136700 0x7f2638136bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 -- 172.21.15.138:0/3821831476 >> v2:172.21.15.138:3300/0 conn(0x7f26380a0820 msgr2=0x7f26380a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 --2- 172.21.15.138:0/3821831476 >> v2:172.21.15.138:3300/0 conn(0x7f26380a0820 0x7f26380a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 monclient: found mon.noname-b 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f26377fe640 1 -- 172.21.15.138:0/3821831476 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263809f5a0 con 0x7f26380a3940 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f0fc6ffd640 1 -- 172.21.15.138:0/985114716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc8070010 msgr2=0x7f0fc8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.034+0000 7f0fc6ffd640 1 --2- 172.21.15.138:0/985114716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc8070010 0x7f0fc8135450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009870 tx=0x7f0fb801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31161a5c7fe2f23 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 1 -- 172.21.15.138:0/985114716 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb80b2070 con 0x7f0fc8070010 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: dump: 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.071 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 1 -- 172.21.15.138:0/985114716 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fb80ad640 con 0x7f0fc8070010 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 5 monclient: authenticate success, global_id 4233 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 4 set_mon_vals no callback set 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 1 -- 172.21.15.138:0/985114716 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb80ada10 con 0x7f0fc8070010 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc5ffb640 10 monclient: dump: 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient: shutdown 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- 172.21.15.138:0/985114716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc8070010 msgr2=0x7f0fc8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 --2- 172.21.15.138:0/985114716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc8070010 0x7f0fc8135450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009870 tx=0x7f0fb801c3f0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- 172.21.15.138:0/985114716 shutdown_connections 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 --2- 172.21.15.138:0/985114716 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0fc809e6e0 0x7f0fc809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 --2- 172.21.15.138:0/985114716 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc8070010 0x7f0fc8135450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- 172.21.15.138:0/985114716 >> 172.21.15.138:0/985114716 conn(0x7f0fc80ab830 msgr2=0x7f0fc80a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- 172.21.15.138:0/985114716 shutdown_connections 2024-09-11T05:07:52.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- 172.21.15.138:0/985114716 wait complete. 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988ddd1640 1 -- 172.21.15.138:0/2921608675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880026410 msgr2=0x7f9880023df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988ddd1640 1 --2- 172.21.15.138:0/2921608675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880026410 0x7f9880023df0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9888000b70 tx=0x7f9888099bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dbbc21c0c9bc66d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 1 -- 172.21.15.138:0/2921608675 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9888125620 con 0x7f9880026410 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: dump: 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.075 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 1 -- 172.21.15.138:0/2921608675 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98881257c0 con 0x7f9880026410 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 1 -- 172.21.15.138:0/2921608675 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9888125bb0 con 0x7f9880026410 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 5 monclient: authenticate success, global_id 4235 2024-09-11T05:07:52.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 4 set_mon_vals no callback set 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988cdcf640 10 monclient: dump: 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.077 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient: monmap: 2024-09-11T05:07:52.078 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.078 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.078 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.078 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.078 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.078 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 librados: starting msgr at 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 librados: starting objecter 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 Processor -- start 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- start start 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 librados: setting wanted keys 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 librados: calling monclient init 2024-09-11T05:07:52.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient: init 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 10 monclient: shutdown 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 -- 172.21.15.138:0/2921608675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880026410 msgr2=0x7f9880023df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 --2- 172.21.15.138:0/2921608675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880026410 0x7f9880023df0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9888000b70 tx=0x7f9888099bc0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 -- 172.21.15.138:0/2921608675 shutdown_connections 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 --2- 172.21.15.138:0/2921608675 >> v2:172.21.15.190:3300/0 conn(0x7f9880006b90 0x7f9880007000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 --2- 172.21.15.138:0/2921608675 >> v2:172.21.15.138:3300/0 conn(0x7f9880024330 0x7f9880006650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 --2- 172.21.15.138:0/2921608675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880026410 0x7f9880023df0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 -- 172.21.15.138:0/2921608675 >> 172.21.15.138:0/2921608675 conn(0x7f9880027530 msgr2=0x7f9880027940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 -- 172.21.15.138:0/2921608675 shutdown_connections 2024-09-11T05:07:52.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 -- 172.21.15.138:0/2921608675 wait complete. 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0fc8070010 0x7f0fc81d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc8070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc809e6e0 0x7f0fc813f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc809e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc81382c0 0x7f0fc813fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc81382c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc81432f0 con 0x7f0fc809e6e0 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc8143170 con 0x7f0fc81382c0 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc8143470 con 0x7f0fc8070010 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fcd831640 10 monclient(hunting): authenticate will time out at 955.479035s 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc809e6e0 0x7f0fc813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc809e6e0 0x7f0fc813f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46272/0 (socket says 172.21.15.138:46272) 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 -- 172.21.15.138:0/684300680 learned_addr learned my addr 172.21.15.138:0/684300680 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient(hunting): get_auth_request con 0x7f0fc809e6e0 auth_method 0 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a86ffd640 1 -- 172.21.15.138:0/1741132804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 msgr2=0x7f1a880a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a86ffd640 1 --2- 172.21.15.138:0/1741132804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 0x7f1a880a3bb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1a780097c0 tx=0x7f1a7801c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8d66813a56b79c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc77fe640 1 --2- 172.21.15.138:0/684300680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc81382c0 0x7f0fc813fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 -- 172.21.15.138:0/684300680 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0fc8070010 msgr2=0x7f0fc81d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 --2- 172.21.15.138:0/684300680 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0fc8070010 0x7f0fc81d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 -- 172.21.15.138:0/684300680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc81382c0 msgr2=0x7f0fc813fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 --2- 172.21.15.138:0/684300680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc81382c0 0x7f0fc813fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 monclient: found mon.noname-a 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc67fc640 1 -- 172.21.15.138:0/684300680 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc813b040 con 0x7f0fc809e6e0 2024-09-11T05:07:52.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc77fe640 10 monclient: get_auth_request con 0x7f0fc81382c0 auth_method 0 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f0fc77fe640 1 --2- 172.21.15.138:0/684300680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0fc81382c0 0x7f0fc813fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 1 -- 172.21.15.138:0/1741132804 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a780b0070 con 0x7f1a880a37d0 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: dump: 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.086 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 1 -- 172.21.15.138:0/1741132804 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a780ab600 con 0x7f1a880a37d0 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 1 -- 172.21.15.138:0/1741132804 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a780ab9d0 con 0x7f1a880a37d0 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a8d59e640 5 monclient: authenticate success, global_id 4236 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a8d59e640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a8d59e640 4 set_mon_vals no callback set 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 10 monclient: monmap: 2024-09-11T05:07:52.088 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f1a85ffb640 10 monclient: dump: 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f26377fe640 1 -- 172.21.15.138:0/3821831476 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 msgr2=0x7f26380a3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f26377fe640 1 --2- 172.21.15.138:0/3821831476 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f26380a3d20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f263806d7d0 tx=0x7f262801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dbfbeba3808fec4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 librados: starting msgr at 2024-09-11T05:07:52.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f988edd3640 1 librados: starting objecter 2024-09-11T05:07:52.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f2636ffd640 1 -- 172.21.15.138:0/3821831476 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26280b0070 con 0x7f26380a3940 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f2636ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f2636ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f2636ffd640 10 monclient: dump: 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f2636ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f2636ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.038+0000 7f2636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2636ffd640 1 -- 172.21.15.138:0/3821831476 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26280ab600 con 0x7f26380a3940 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2636ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2636ffd640 1 -- 172.21.15.138:0/3821831476 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26280ab9d0 con 0x7f26380a3940 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2636ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2636ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2636ffd640 10 monclient: dump: 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.093 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.094 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.094 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.094 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.094 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 5 monclient: authenticate success, global_id 4238 2024-09-11T05:07:52.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 4 set_mon_vals no callback set 2024-09-11T05:07:52.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 Processor -- start 2024-09-11T05:07:52.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 -- start start 2024-09-11T05:07:52.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 librados: setting wanted keys 2024-09-11T05:07:52.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 librados: calling monclient init 2024-09-11T05:07:52.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient: init 2024-09-11T05:07:52.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient: shutdown 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- 172.21.15.138:0/1741132804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 msgr2=0x7f1a880a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 --2- 172.21.15.138:0/1741132804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 0x7f1a880a3bb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1a780097c0 tx=0x7f1a7801c310 comp rx=0 tx=0).stop 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- 172.21.15.138:0/1741132804 shutdown_connections 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 --2- 172.21.15.138:0/1741132804 >> v2:172.21.15.190:3300/0 conn(0x7f1a8809d8e0 0x7f1a880a40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 --2- 172.21.15.138:0/1741132804 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 0x7f1a880a3bb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- 172.21.15.138:0/1741132804 >> 172.21.15.138:0/1741132804 conn(0x7f1a880a4a90 msgr2=0x7f1a880a4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- 172.21.15.138:0/1741132804 shutdown_connections 2024-09-11T05:07:52.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- 172.21.15.138:0/1741132804 wait complete. 2024-09-11T05:07:52.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:07:52.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9880006b90 0x7f988014edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient(hunting): picked mon.noname-a con 0x7f9880006b90 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient: shutdown 2024-09-11T05:07:52.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- 172.21.15.138:0/3821831476 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 msgr2=0x7f26380a3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 --2- 172.21.15.138:0/3821831476 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f26380a3d20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f263806d7d0 tx=0x7f262801c3e0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880024330 0x7f988014f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient(hunting): picked mon.noname-c con 0x7f9880024330 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- 172.21.15.138:0/3821831476 shutdown_connections 2024-09-11T05:07:52.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 --2- 172.21.15.138:0/3821831476 >> v2:172.21.15.190:3300/0 conn(0x7f2638136700 0x7f2638136bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 --2- 172.21.15.138:0/3821831476 >> v2:172.21.15.138:3300/0 conn(0x7f26380a0820 0x7f26380a4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 --2- 172.21.15.138:0/3821831476 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f26380a3d20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- 172.21.15.138:0/3821831476 >> 172.21.15.138:0/3821831476 conn(0x7f26380a5030 msgr2=0x7f2638063640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- 172.21.15.138:0/3821831476 shutdown_connections 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9880026410 0x7f988014f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient(hunting): picked mon.noname-b con 0x7f9880026410 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9880024890 con 0x7f9880006b90 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9880024710 con 0x7f9880026410 2024-09-11T05:07:52.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f988001e550 con 0x7f9880024330 2024-09-11T05:07:52.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient(hunting): authenticate will time out at 955.480773s 2024-09-11T05:07:52.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9880006b90 0x7f988014edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880024330 0x7f988014f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- 172.21.15.138:0/3821831476 wait complete. 2024-09-11T05:07:52.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9880006b90 0x7f988014edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46284/0 (socket says 172.21.15.138:46284) 2024-09-11T05:07:52.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880024330 0x7f988014f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58762/0 (socket says 172.21.15.138:58762) 2024-09-11T05:07:52.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 1 -- 172.21.15.138:0/2006529604 learned_addr learned my addr 172.21.15.138:0/2006529604 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 10 monclient(hunting): get_auth_request con 0x7f9880006b90 auth_method 0 2024-09-11T05:07:52.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient(hunting): get_auth_request con 0x7f9880024330 auth_method 0 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988ddd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988e5d2640 1 --2- 172.21.15.138:0/2006529604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9880026410 0x7f988014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 -- 172.21.15.138:0/2006529604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9880026410 msgr2=0x7f988014f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 --2- 172.21.15.138:0/2006529604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9880026410 0x7f988014f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 -- 172.21.15.138:0/2006529604 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9880006b90 msgr2=0x7f988014edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 --2- 172.21.15.138:0/2006529604 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9880006b90 0x7f988014edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 -- 172.21.15.138:0/2006529604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98800bf360 con 0x7f9880024330 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient: monmap: 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.104 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 librados: starting msgr at 2024-09-11T05:07:52.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 librados: starting objecter 2024-09-11T05:07:52.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 Processor -- start 2024-09-11T05:07:52.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient: monmap: 2024-09-11T05:07:52.105 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- start start 2024-09-11T05:07:52.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 librados: setting wanted keys 2024-09-11T05:07:52.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 librados: calling monclient init 2024-09-11T05:07:52.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient: init 2024-09-11T05:07:52.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fc67fc640 1 -- 172.21.15.138:0/684300680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc809e6e0 msgr2=0x7f0fc813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fc67fc640 1 --2- 172.21.15.138:0/684300680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc809e6e0 0x7f0fc813f880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00e900 tx=0x7f0fbc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 client.?.objecter ms_handle_connect 0x7f0fc809e6e0 2024-09-11T05:07:52.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 librados: starting msgr at 2024-09-11T05:07:52.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 librados: starting objecter 2024-09-11T05:07:52.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc017070 con 0x7f0fc809e6e0 2024-09-11T05:07:52.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 Processor -- start 2024-09-11T05:07:52.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:52.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: dump: 2024-09-11T05:07:52.109 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fbc00cc10 con 0x7f0fc809e6e0 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 5 monclient: authenticate success, global_id 4239 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 10 monclient: _renew_subs 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc8140300 con 0x7f0fc809e6e0 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc013470 con 0x7f0fc809e6e0 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: dump: 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.111 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 10 monclient: _renew_subs 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc813ad60 con 0x7f0fc809e6e0 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 1 librados: init done 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- start start 2024-09-11T05:07:52.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 librados: setting wanted keys 2024-09-11T05:07:52.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 librados: calling monclient init 2024-09-11T05:07:52.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient: init 2024-09-11T05:07:52.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a8806d470 0x7f1a881d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient(hunting): picked mon.noname-c con 0x7f1a8806d470 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a8809d8e0 0x7f1a881376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient(hunting): picked mon.noname-b con 0x7f1a8809d8e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 0x7f1a8811e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f263809ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient(hunting): picked mon.noname-c con 0x7f26380a3940 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2638136700 0x7f263809cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient(hunting): picked mon.noname-a con 0x7f2638136700 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f263809d490 0x7f26381d54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient(hunting): picked mon.noname-b con 0x7f263809d490 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2638064130 con 0x7f2638136700 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2638063fb0 con 0x7f263809d490 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2638140170 con 0x7f26380a3940 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f263deef640 10 monclient(hunting): authenticate will time out at 955.482833s 2024-09-11T05:07:52.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f263809ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f263809ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58772/0 (socket says 172.21.15.138:58772) 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 -- 172.21.15.138:0/996474431 learned_addr learned my addr 172.21.15.138:0/996474431 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient(hunting): get_auth_request con 0x7f26380a3940 auth_method 0 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 1 --2- 172.21.15.138:0/996474431 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2638136700 0x7f263809cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2637fff640 1 --2- 172.21.15.138:0/996474431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f263809d490 0x7f26381d54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 10 monclient(hunting): get_auth_request con 0x7f2638136700 auth_method 0 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fc57fa640 4 set_mon_vals no callback set 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 10 client.4239.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 15 client.4239.objecter _session_command_op_assign -1 1 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 20 client.4239.objecter _get_session osd=-1 returning homeless 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0fc8135450 con 0x7f0fc809e6e0 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a8806d470 0x7f1a881d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient(hunting): picked mon.noname-a con 0x7f1a880a37d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a8806d470 0x7f1a881d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58764/0 (socket says 172.21.15.138:58764) 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1a88142dc0 con 0x7f1a880a37d0 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 -- 172.21.15.138:0/2183307561 learned_addr learned my addr 172.21.15.138:0/2183307561 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1a88142c40 con 0x7f1a8809d8e0 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1a88142f40 con 0x7f1a8806d470 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a8d59e640 10 monclient(hunting): authenticate will time out at 955.482580s 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient(hunting): get_auth_request con 0x7f1a8806d470 auth_method 0 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a877fe640 1 --2- 172.21.15.138:0/2183307561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 0x7f1a8811e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a877fe640 10 monclient(hunting): get_auth_request con 0x7f1a880a37d0 auth_method 0 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 -- 172.21.15.138:0/2183307561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a8809d8e0 msgr2=0x7f1a881376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 --2- 172.21.15.138:0/2183307561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a8809d8e0 0x7f1a881376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 -- 172.21.15.138:0/2183307561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 msgr2=0x7f1a8811e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 --2- 172.21.15.138:0/2183307561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a880a37d0 0x7f1a8811e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f1a86ffd640 1 -- 172.21.15.138:0/2183307561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a881d10f0 con 0x7f1a8806d470 2024-09-11T05:07:52.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2637fff640 10 monclient(hunting): get_auth_request con 0x7f263809d490 auth_method 0 2024-09-11T05:07:52.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2637fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f2637fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f0fbc024060 con 0x7f0fc809e6e0 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 -- 172.21.15.138:0/996474431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f263809d490 msgr2=0x7f26381d54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 --2- 172.21.15.138:0/996474431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f263809d490 0x7f26381d54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 -- 172.21.15.138:0/996474431 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2638136700 msgr2=0x7f263809cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 --2- 172.21.15.138:0/996474431 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2638136700 0x7f263809cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f26377fe640 1 -- 172.21.15.138:0/996474431 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263811eb30 con 0x7f26380a3940 2024-09-11T05:07:52.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f262effd640 1 --2- 172.21.15.138:0/996474431 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2638136700 0x7f263809cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:52.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 -- 172.21.15.138:0/2006529604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880024330 msgr2=0x7f988014f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988d5d0640 1 --2- 172.21.15.138:0/2006529604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880024330 0x7f988014f330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f988400c7f0 tx=0x7f988400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 client.?.objecter ms_handle_connect 0x7f9880024330 2024-09-11T05:07:52.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 --2- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fb403cef0 0x7f0fb403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884015070 con 0x7f9880024330 2024-09-11T05:07:52.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f0fbc053120 con 0x7f0fc809e6e0 2024-09-11T05:07:52.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 client.4239.objecter ms_dispatch 0x7f0fc8000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: dump: 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9884004220 con 0x7f9880024330 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884007830 con 0x7f9880024330 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 3 client.4239.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 3 client.4239.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 5 monclient: authenticate success, global_id 4241 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987e7fc640 10 monclient: dump: 2024-09-11T05:07:52.126 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient: _renew_subs 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9880021780 con 0x7f9880024330 2024-09-11T05:07:52.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f987ffff640 4 set_mon_vals no callback set 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 client.4239.objecter checking command 1 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 --2- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6809/3046884208 conn(0x7f0fb4042a00 0x7f0fb4044e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 20 client.4239.objecter _get_session s=0x7f0fb4042350 osd=3 3 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 20 client.4239.objecter put_session s=0x7f0fb4042350 osd=3 3 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 15 client.4239.objecter _session_command_op_remove -1 1 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 20 client.4239.objecter _get_session s=0x7f0fb4042350 osd=3 3 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 20 client.4239.objecter get_session s=0x7f0fb4042350 osd=3 3 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 15 client.4239.objecter _session_command_op_assign 3 1 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 20 client.4239.objecter put_session s=0x7f0fb4042350 osd=3 4 2024-09-11T05:07:52.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 client.4239.objecter _send_command 1 2024-09-11T05:07:52.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient: _renew_subs 2024-09-11T05:07:52.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 --> v2:172.21.15.190:6809/3046884208 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fbc017e50 con 0x7f0fb4042a00 2024-09-11T05:07:52.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 20 client.4239.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f0fbc055050 con 0x7f0fc809e6e0 2024-09-11T05:07:52.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f0faffff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T05:07:52.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f98800bdb60 con 0x7f9880024330 2024-09-11T05:07:52.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.042+0000 7f988edd3640 1 librados: init done 2024-09-11T05:07:52.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f0fc6ffd640 1 --2- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fb403cef0 0x7f0fb403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f0fc6ffd640 10 monclient: get_auth_request con 0x7f0fb403cef0 auth_method 0 2024-09-11T05:07:52.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f0fc77fe640 1 --2- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6809/3046884208 conn(0x7f0fb4042a00 0x7f0fb4044e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f0fc77fe640 10 monclient: get_auth_request con 0x7f0fb4042a00 auth_method 0 2024-09-11T05:07:52.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988edd3640 10 client.4241.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988edd3640 15 client.4241.objecter _session_command_op_assign -1 1 2024-09-11T05:07:52.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988edd3640 20 client.4241.objecter _get_session osd=-1 returning homeless 2024-09-11T05:07:52.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988edd3640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988edd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f988001f710 con 0x7f9880024330 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f9884007a10 con 0x7f9880024330 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 1 --2- 172.21.15.138:0/2006529604 >> v2:172.21.15.190:6800/1562782189 conn(0x7f986c03cf20 0x7f986c03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f988404f1a0 con 0x7f9880024330 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 10 client.4241.objecter ms_dispatch 0x7f9888000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 3 client.4241.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 3 client.4241.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 10 client.4241.objecter checking command 1 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 1 --2- 172.21.15.138:0/2006529604 >> v2:172.21.15.138:6804/81047329 conn(0x7f986c042ab0 0x7f986c044ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 20 client.4241.objecter _get_session s=0x7f986c042400 osd=2 3 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 20 client.4241.objecter put_session s=0x7f986c042400 osd=2 3 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 15 client.4241.objecter _session_command_op_remove -1 1 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 20 client.4241.objecter _get_session s=0x7f986c042400 osd=2 3 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 20 client.4241.objecter get_session s=0x7f986c042400 osd=2 3 2024-09-11T05:07:52.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 15 client.4241.objecter _session_command_op_assign 2 1 2024-09-11T05:07:52.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 20 client.4241.objecter put_session s=0x7f986c042400 osd=2 4 2024-09-11T05:07:52.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 10 client.4241.objecter _send_command 1 2024-09-11T05:07:52.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 --> v2:172.21.15.138:6804/81047329 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f986c045580 con 0x7f986c042ab0 2024-09-11T05:07:52.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 20 client.4241.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f9884056050 con 0x7f9880024330 2024-09-11T05:07:52.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f987e7fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T05:07:52.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988e5d2640 1 --2- 172.21.15.138:0/2006529604 >> v2:172.21.15.138:6804/81047329 conn(0x7f986c042ab0 0x7f986c044ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988e5d2640 10 monclient: get_auth_request con 0x7f986c042ab0 auth_method 0 2024-09-11T05:07:52.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a86ffd640 1 -- 172.21.15.138:0/2183307561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a8806d470 msgr2=0x7f1a881d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a86ffd640 1 --2- 172.21.15.138:0/2183307561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a8806d470 0x7f1a881d01e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1a7801cbb0 tx=0x7f1a780190b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 client.?.objecter ms_handle_connect 0x7f1a8806d470 2024-09-11T05:07:52.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a780b8020 con 0x7f1a8806d470 2024-09-11T05:07:52.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: dump: 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.136 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a780b4810 con 0x7f1a8806d470 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 5 monclient: authenticate success, global_id 4244 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 10 monclient: _renew_subs 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a88121da0 con 0x7f1a8806d470 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a780b0040 con 0x7f1a8806d470 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: dump: 2024-09-11T05:07:52.137 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 10 monclient: _renew_subs 2024-09-11T05:07:52.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8811fd20 con 0x7f1a8806d470 2024-09-11T05:07:52.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 1 librados: init done 2024-09-11T05:07:52.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a857fa640 4 set_mon_vals no callback set 2024-09-11T05:07:52.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f26377fe640 1 -- 172.21.15.138:0/996474431 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 msgr2=0x7f263809ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f26377fe640 1 --2- 172.21.15.138:0/996474431 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f263809ca10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f26280ae040 tx=0x7f262801ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 client.?.objecter ms_handle_connect 0x7f26380a3940 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26280ba020 con 0x7f26380a3940 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: dump: 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.140 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26280b6630 con 0x7f26380a3940 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 5 monclient: authenticate success, global_id 4247 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 10 monclient: _renew_subs 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 10 client.4244.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 15 client.4244.objecter _session_command_op_assign -1 1 2024-09-11T05:07:52.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 20 client.4244.objecter _get_session osd=-1 returning homeless 2024-09-11T05:07:52.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1a880a78c0 con 0x7f1a8806d470 2024-09-11T05:07:52.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2638074030 con 0x7f26380a3940 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 10 monclient: _renew_subs 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f263811f3b0 con 0x7f26380a3940 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 1 librados: init done 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26280b0040 con 0x7f26380a3940 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: dump: 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.143 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f26367fc640 4 set_mon_vals no callback set 2024-09-11T05:07:52.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1a780b4c40 con 0x7f1a8806d470 2024-09-11T05:07:52.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f0faffff640 10 client.4239.objecter ms_handle_connect 0x7f0fb403cef0 2024-09-11T05:07:52.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f0fc6ffd640 1 --2- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fb403cef0 0x7f0fb403f3b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0fc80a7a00 tx=0x7f0fb8004740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988ddd1640 1 --2- 172.21.15.138:0/2006529604 >> v2:172.21.15.190:6800/1562782189 conn(0x7f986c03cf20 0x7f986c03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f988ddd1640 10 monclient: get_auth_request con 0x7f986c03cf20 auth_method 0 2024-09-11T05:07:52.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 --2- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a7403cea0 0x7f1a7403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f1a780e92f0 con 0x7f1a8806d470 2024-09-11T05:07:52.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 client.4244.objecter ms_dispatch 0x7f1a88000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 3 client.4244.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 3 client.4244.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 client.4244.objecter checking command 1 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 --2- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1a740429b0 0x7f1a74044df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 20 client.4244.objecter _get_session s=0x7f1a74042300 osd=4 3 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 20 client.4244.objecter put_session s=0x7f1a74042300 osd=4 3 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 15 client.4244.objecter _session_command_op_remove -1 1 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 20 client.4244.objecter _get_session s=0x7f1a74042300 osd=4 3 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 20 client.4244.objecter get_session s=0x7f1a74042300 osd=4 3 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 15 client.4244.objecter _session_command_op_assign 4 1 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 20 client.4244.objecter put_session s=0x7f1a74042300 osd=4 4 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 client.4244.objecter _send_command 1 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 --> v2:172.21.15.190:6801/1118550024 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a780ace50 con 0x7f1a740429b0 2024-09-11T05:07:52.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 20 client.4244.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f1a780f0050 con 0x7f1a8806d470 2024-09-11T05:07:52.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a6f7fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T05:07:52.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a867fc640 1 --2- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a7403cea0 0x7f1a7403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a867fc640 10 monclient: get_auth_request con 0x7f1a7403cea0 auth_method 0 2024-09-11T05:07:52.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f26280b3030 con 0x7f26380a3940 2024-09-11T05:07:52.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 10 client.4247.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 15 client.4247.objecter _session_command_op_assign -1 1 2024-09-11T05:07:52.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 20 client.4247.objecter _get_session osd=-1 returning homeless 2024-09-11T05:07:52.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f261c002920 con 0x7f26380a3940 2024-09-11T05:07:52.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a877fe640 1 --2- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1a740429b0 0x7f1a74044df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f1a877fe640 10 monclient: get_auth_request con 0x7f1a740429b0 auth_method 0 2024-09-11T05:07:52.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 --2- 172.21.15.138:0/996474431 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261803cef0 0x7f261803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f26280e9220 con 0x7f26380a3940 2024-09-11T05:07:52.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 client.4247.objecter ms_dispatch 0x7f2638000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 3 client.4247.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 3 client.4247.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 client.4247.objecter checking command 1 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 --2- 172.21.15.138:0/996474431 >> v2:172.21.15.138:6808/4002974512 conn(0x7f2618042a00 0x7f2618044e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 20 client.4247.objecter _get_session s=0x7f2618042350 osd=1 3 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 20 client.4247.objecter put_session s=0x7f2618042350 osd=1 3 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 15 client.4247.objecter _session_command_op_remove -1 1 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 20 client.4247.objecter _get_session s=0x7f2618042350 osd=1 3 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 20 client.4247.objecter get_session s=0x7f2618042350 osd=1 3 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 15 client.4247.objecter _session_command_op_assign 1 1 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 20 client.4247.objecter put_session s=0x7f2618042350 osd=1 4 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 client.4247.objecter _send_command 1 2024-09-11T05:07:52.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 --> v2:172.21.15.138:6808/4002974512 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26280bdec0 con 0x7f2618042a00 2024-09-11T05:07:52.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 20 client.4247.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f26280f0050 con 0x7f26380a3940 2024-09-11T05:07:52.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.046+0000 7f2634ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T05:07:52.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f2637fff640 1 --2- 172.21.15.138:0/996474431 >> v2:172.21.15.138:6808/4002974512 conn(0x7f2618042a00 0x7f2618044e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f2637fff640 10 monclient: get_auth_request con 0x7f2618042a00 auth_method 0 2024-09-11T05:07:52.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f0faffff640 10 client.4239.objecter ms_handle_connect 0x7f0fb4042a00 2024-09-11T05:07:52.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f0fc77fe640 1 --2- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6809/3046884208 conn(0x7f0fb4042a00 0x7f0fb4044e40 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f988e5d2640 1 --2- 172.21.15.138:0/2006529604 >> v2:172.21.15.138:6804/81047329 conn(0x7f986c042ab0 0x7f986c044ef0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f987e7fc640 10 client.4241.objecter ms_handle_connect 0x7f986c042ab0 2024-09-11T05:07:52.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f987e7fc640 10 client.4241.objecter ms_handle_connect 0x7f986c03cf20 2024-09-11T05:07:52.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f988ddd1640 1 --2- 172.21.15.138:0/2006529604 >> v2:172.21.15.190:6800/1562782189 conn(0x7f986c03cf20 0x7f986c03f3e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f98880982c0 tx=0x7f9888071030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f262effd640 1 --2- 172.21.15.138:0/996474431 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261803cef0 0x7f261803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f262effd640 10 monclient: get_auth_request con 0x7f261803cef0 auth_method 0 2024-09-11T05:07:52.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f1a6f7fe640 10 client.4244.objecter ms_handle_connect 0x7f1a7403cea0 2024-09-11T05:07:52.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f1a867fc640 1 --2- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a7403cea0 0x7f1a7403f360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1a88140920 tx=0x7f1a7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 <== osd.2 v2:172.21.15.138:6804/81047329 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f986c045580 con 0x7f986c042ab0 2024-09-11T05:07:52.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f987e7fc640 10 client.4241.objecter ms_dispatch 0x7f9888000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:07:52.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f987e7fc640 10 client.4241.objecter _finish_command 1 = system:0 2024-09-11T05:07:52.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f987e7fc640 20 client.4241.objecter put_session s=0x7f986c042400 osd=2 4 2024-09-11T05:07:52.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f987e7fc640 15 client.4241.objecter _session_command_op_remove 2 1 2024-09-11T05:07:52.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 <== osd.3 v2:172.21.15.190:6809/3046884208 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fbc017e50 con 0x7f0fb4042a00 2024-09-11T05:07:52.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f0faffff640 10 client.4239.objecter ms_dispatch 0x7f0fc8000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:07:52.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f0faffff640 10 client.4239.objecter _finish_command 1 = system:0 2024-09-11T05:07:52.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f0faffff640 20 client.4239.objecter put_session s=0x7f0fb4042350 osd=3 4 2024-09-11T05:07:52.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.050+0000 7f0faffff640 15 client.4239.objecter _session_command_op_remove 3 1 2024-09-11T05:07:52.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f1a6f7fe640 10 client.4244.objecter ms_handle_connect 0x7f1a740429b0 2024-09-11T05:07:52.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f1a877fe640 1 --2- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1a740429b0 0x7f1a74044df0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f2634ff9640 10 client.4247.objecter ms_handle_connect 0x7f2618042a00 2024-09-11T05:07:52.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f2637fff640 1 --2- 172.21.15.138:0/996474431 >> v2:172.21.15.138:6808/4002974512 conn(0x7f2618042a00 0x7f2618044e40 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f2634ff9640 10 client.4247.objecter ms_handle_connect 0x7f261803cef0 2024-09-11T05:07:52.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f262effd640 1 --2- 172.21.15.138:0/996474431 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261803cef0 0x7f261803f3b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f263809e970 tx=0x7f2620009110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 <== osd.4 v2:172.21.15.190:6801/1118550024 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1a780ace50 con 0x7f1a740429b0 2024-09-11T05:07:52.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f1a6f7fe640 10 client.4244.objecter ms_dispatch 0x7f1a88000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:07:52.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f1a6f7fe640 10 client.4244.objecter _finish_command 1 = system:0 2024-09-11T05:07:52.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f1a6f7fe640 20 client.4244.objecter put_session s=0x7f1a74042300 osd=4 4 2024-09-11T05:07:52.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f1a6f7fe640 15 client.4244.objecter _session_command_op_remove 4 1 2024-09-11T05:07:52.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 <== osd.1 v2:172.21.15.138:6808/4002974512 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f26280bdec0 con 0x7f2618042a00 2024-09-11T05:07:52.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f2634ff9640 10 client.4247.objecter ms_dispatch 0x7f2638000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:07:52.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f2634ff9640 10 client.4247.objecter _finish_command 1 = system:0 2024-09-11T05:07:52.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f2634ff9640 20 client.4247.objecter put_session s=0x7f2618042350 osd=1 4 2024-09-11T05:07:52.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.054+0000 7f2634ff9640 15 client.4247.objecter _session_command_op_remove 1 1 2024-09-11T05:07:52.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 10 client.4239.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 15 client.4239.objecter _session_command_op_assign -1 2 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 20 client.4239.objecter _get_session s=0x7f0fb4042350 osd=3 3 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 20 client.4239.objecter put_session s=0x7f0fb4042350 osd=3 3 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 20 client.4239.objecter _get_session s=0x7f0fb4042350 osd=3 3 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 15 client.4239.objecter _session_command_op_remove -1 2 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 20 client.4239.objecter get_session s=0x7f0fb4042350 osd=3 3 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 15 client.4239.objecter _session_command_op_assign 3 2 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 20 client.4239.objecter put_session s=0x7f0fb4042350 osd=3 4 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 10 client.4239.objecter _send_command 2 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 --> v2:172.21.15.190:6809/3046884208 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0fc80a2330 con 0x7f0fb4042a00 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0faffff640 1 -- 172.21.15.138:0/684300680 <== osd.3 v2:172.21.15.190:6809/3046884208 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0fc80a2330 con 0x7f0fb4042a00 2024-09-11T05:07:52.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0faffff640 10 client.4239.objecter ms_dispatch 0x7f0fc8000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:07:52.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0faffff640 10 client.4239.objecter _finish_command 2 = system:0 2024-09-11T05:07:52.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0faffff640 20 client.4239.objecter put_session s=0x7f0fb4042350 osd=3 4 2024-09-11T05:07:52.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0faffff640 15 client.4239.objecter _session_command_op_remove 3 2 2024-09-11T05:07:52.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 10 librados: watch_flush enter 2024-09-11T05:07:52.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 10 librados: watch_flush exit 2024-09-11T05:07:52.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 10 client.4239.objecter close_session for osd.3 2024-09-11T05:07:52.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6809/3046884208 conn(0x7f0fb4042a00 msgr2=0x7f0fb4044e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 --2- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6809/3046884208 conn(0x7f0fb4042a00 0x7f0fb4044e40 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 20 client.4239.objecter put_session s=0x7f0fb4042350 osd=3 1 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 20 client.4239.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 10 client.4239.objecter successfully canceled tick 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fb403cef0 msgr2=0x7f0fb403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 --2- 172.21.15.138:0/684300680 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fb403cef0 0x7f0fb403f3b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0fc80a7a00 tx=0x7f0fb8004740 comp rx=0 tx=0).stop 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 10 monclient: shutdown 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc809e6e0 msgr2=0x7f0fc813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 --2- 172.21.15.138:0/684300680 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0fc809e6e0 0x7f0fc813f880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00e900 tx=0x7f0fbc00edc0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fc77fe640 1 -- 172.21.15.138:0/684300680 reap_dead start 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 shutdown_connections 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 >> 172.21.15.138:0/684300680 conn(0x7f0fc80ab830 msgr2=0x7f0fc80a97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 shutdown_connections 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 -- 172.21.15.138:0/684300680 wait complete. 2024-09-11T05:07:52.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.066+0000 7f0fcd831640 1 librados: shutdown 2024-09-11T05:07:52.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 10 client.4247.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T05:07:52.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 15 client.4247.objecter _session_command_op_assign -1 2 2024-09-11T05:07:52.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 20 client.4247.objecter _get_session s=0x7f2618042350 osd=1 3 2024-09-11T05:07:52.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 20 client.4247.objecter put_session s=0x7f2618042350 osd=1 3 2024-09-11T05:07:52.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 20 client.4247.objecter _get_session s=0x7f2618042350 osd=1 3 2024-09-11T05:07:52.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 15 client.4247.objecter _session_command_op_remove -1 2 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 20 client.4247.objecter get_session s=0x7f2618042350 osd=1 3 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 15 client.4247.objecter _session_command_op_assign 1 2 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 20 client.4247.objecter put_session s=0x7f2618042350 osd=1 4 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 10 client.4247.objecter _send_command 2 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 --> v2:172.21.15.138:6808/4002974512 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f261c001230 con 0x7f2618042a00 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f2634ff9640 1 -- 172.21.15.138:0/996474431 <== osd.1 v2:172.21.15.138:6808/4002974512 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f261c001230 con 0x7f2618042a00 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f2634ff9640 10 client.4247.objecter ms_dispatch 0x7f2638000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f2634ff9640 10 client.4247.objecter _finish_command 2 = system:0 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f2634ff9640 20 client.4247.objecter put_session s=0x7f2618042350 osd=1 4 2024-09-11T05:07:52.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f2634ff9640 15 client.4247.objecter _session_command_op_remove 1 2 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 10 librados: watch_flush enter 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 10 librados: watch_flush exit 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 10 client.4247.objecter close_session for osd.1 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 >> v2:172.21.15.138:6808/4002974512 conn(0x7f2618042a00 msgr2=0x7f2618044e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 --2- 172.21.15.138:0/996474431 >> v2:172.21.15.138:6808/4002974512 conn(0x7f2618042a00 0x7f2618044e40 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 20 client.4247.objecter put_session s=0x7f2618042350 osd=1 1 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 20 client.4247.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 10 client.4247.objecter successfully canceled tick 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261803cef0 msgr2=0x7f261803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 --2- 172.21.15.138:0/996474431 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261803cef0 0x7f261803f3b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f263809e970 tx=0x7f2620009110 comp rx=0 tx=0).stop 2024-09-11T05:07:52.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 10 monclient: shutdown 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 msgr2=0x7f263809ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 --2- 172.21.15.138:0/996474431 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f26380a3940 0x7f263809ca10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f26280ae040 tx=0x7f262801ac20 comp rx=0 tx=0).stop 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f2637fff640 1 -- 172.21.15.138:0/996474431 reap_dead start 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 shutdown_connections 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.070+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 >> 172.21.15.138:0/996474431 conn(0x7f26380a5030 msgr2=0x7f2638072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 shutdown_connections 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f263deef640 1 -- 172.21.15.138:0/996474431 wait complete. 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f263deef640 1 librados: shutdown 2024-09-11T05:07:52.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 10 client.4241.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 15 client.4241.objecter _session_command_op_assign -1 2 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 20 client.4241.objecter _get_session s=0x7f986c042400 osd=2 3 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 20 client.4241.objecter put_session s=0x7f986c042400 osd=2 3 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 20 client.4241.objecter _get_session s=0x7f986c042400 osd=2 3 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 15 client.4241.objecter _session_command_op_remove -1 2 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 20 client.4241.objecter get_session s=0x7f986c042400 osd=2 3 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 15 client.4241.objecter _session_command_op_assign 2 2 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 20 client.4241.objecter put_session s=0x7f986c042400 osd=2 4 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 10 client.4241.objecter _send_command 2 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 --> v2:172.21.15.138:6804/81047329 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f988002a480 con 0x7f986c042ab0 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f987e7fc640 1 -- 172.21.15.138:0/2006529604 <== osd.2 v2:172.21.15.138:6804/81047329 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f988002a480 con 0x7f986c042ab0 2024-09-11T05:07:52.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f987e7fc640 10 client.4241.objecter ms_dispatch 0x7f9888000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:07:52.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f987e7fc640 10 client.4241.objecter _finish_command 2 = system:0 2024-09-11T05:07:52.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f987e7fc640 20 client.4241.objecter put_session s=0x7f986c042400 osd=2 4 2024-09-11T05:07:52.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.074+0000 7f987e7fc640 15 client.4241.objecter _session_command_op_remove 2 2 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 10 librados: watch_flush enter 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 10 librados: watch_flush exit 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 10 client.4241.objecter close_session for osd.2 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 >> v2:172.21.15.138:6804/81047329 conn(0x7f986c042ab0 msgr2=0x7f986c044ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 --2- 172.21.15.138:0/2006529604 >> v2:172.21.15.138:6804/81047329 conn(0x7f986c042ab0 0x7f986c044ef0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 20 client.4241.objecter put_session s=0x7f986c042400 osd=2 1 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 20 client.4241.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 10 client.4241.objecter successfully canceled tick 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 >> v2:172.21.15.190:6800/1562782189 conn(0x7f986c03cf20 msgr2=0x7f986c03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 --2- 172.21.15.138:0/2006529604 >> v2:172.21.15.190:6800/1562782189 conn(0x7f986c03cf20 0x7f986c03f3e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f98880982c0 tx=0x7f9888071030 comp rx=0 tx=0).stop 2024-09-11T05:07:52.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 10 monclient: shutdown 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880024330 msgr2=0x7f988014f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 --2- 172.21.15.138:0/2006529604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9880024330 0x7f988014f330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f988400c7f0 tx=0x7f988400ccb0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988e5d2640 1 -- 172.21.15.138:0/2006529604 reap_dead start 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 shutdown_connections 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 >> 172.21.15.138:0/2006529604 conn(0x7f9880027530 msgr2=0x7f9880029760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 shutdown_connections 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 -- 172.21.15.138:0/2006529604 wait complete. 2024-09-11T05:07:52.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f988edd3640 1 librados: shutdown 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 10 client.4244.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 15 client.4244.objecter _session_command_op_assign -1 2 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 20 client.4244.objecter _get_session s=0x7f1a74042300 osd=4 3 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 20 client.4244.objecter put_session s=0x7f1a74042300 osd=4 3 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 20 client.4244.objecter _get_session s=0x7f1a74042300 osd=4 3 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 15 client.4244.objecter _session_command_op_remove -1 2 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 20 client.4244.objecter get_session s=0x7f1a74042300 osd=4 3 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 15 client.4244.objecter _session_command_op_assign 4 2 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 20 client.4244.objecter put_session s=0x7f1a74042300 osd=4 4 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 10 client.4244.objecter _send_command 2 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.078+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 --> v2:172.21.15.190:6801/1118550024 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1a881385b0 con 0x7f1a740429b0 2024-09-11T05:07:52.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a6f7fe640 1 -- 172.21.15.138:0/2183307561 <== osd.4 v2:172.21.15.190:6801/1118550024 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1a881385b0 con 0x7f1a740429b0 2024-09-11T05:07:52.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a6f7fe640 10 client.4244.objecter ms_dispatch 0x7f1a88000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:07:52.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a6f7fe640 10 client.4244.objecter _finish_command 2 = system:0 2024-09-11T05:07:52.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a6f7fe640 20 client.4244.objecter put_session s=0x7f1a74042300 osd=4 4 2024-09-11T05:07:52.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a6f7fe640 15 client.4244.objecter _session_command_op_remove 4 2 2024-09-11T05:07:52.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 10 librados: watch_flush enter 2024-09-11T05:07:52.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 10 librados: watch_flush exit 2024-09-11T05:07:52.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 10 client.4244.objecter close_session for osd.4 2024-09-11T05:07:52.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1a740429b0 msgr2=0x7f1a74044df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 --2- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1a740429b0 0x7f1a74044df0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 20 client.4244.objecter put_session s=0x7f1a74042300 osd=4 1 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 20 client.4244.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 10 client.4244.objecter successfully canceled tick 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a7403cea0 msgr2=0x7f1a7403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 --2- 172.21.15.138:0/2183307561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a7403cea0 0x7f1a7403f360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1a88140920 tx=0x7f1a7c008040 comp rx=0 tx=0).stop 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 10 monclient: shutdown 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a8806d470 msgr2=0x7f1a881d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 --2- 172.21.15.138:0/2183307561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a8806d470 0x7f1a881d01e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1a7801cbb0 tx=0x7f1a780190b0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a877fe640 1 -- 172.21.15.138:0/2183307561 reap_dead start 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 shutdown_connections 2024-09-11T05:07:52.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 >> 172.21.15.138:0/2183307561 conn(0x7f1a880a4a90 msgr2=0x7f1a880a6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 shutdown_connections 2024-09-11T05:07:52.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 -- 172.21.15.138:0/2183307561 wait complete. 2024-09-11T05:07:52.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.082+0000 7f1a8d59e640 1 librados: shutdown 2024-09-11T05:07:52.176 INFO:teuthology.orchestra.run.smithi138.stdout:51539607558 2024-09-11T05:07:52.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T05:07:52.178 INFO:teuthology.orchestra.run.smithi138.stdout:47244640261 2024-09-11T05:07:52.178 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T05:07:52.179 INFO:teuthology.orchestra.run.smithi138.stdout:47244640260 2024-09-11T05:07:52.179 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T05:07:52.180 INFO:teuthology.orchestra.run.smithi138.stdout:47244640260 2024-09-11T05:07:52.181 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T05:07:52.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient: monmap: 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.199668+0000 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.199668+0000 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.198 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 1 Processor -- start 2024-09-11T05:07:52.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 1 -- start start 2024-09-11T05:07:52.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient: init 2024-09-11T05:07:52.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f527c070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f527c09e6e0 0x7f527c13d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient(hunting): picked mon.noname-c con 0x7f527c09e6e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f527c09eb10 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f527c135b60 con 0x7f527c09eb10 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f527c0a8990 con 0x7f527c070010 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f527c0703f0 con 0x7f527c09e6e0 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5280fbe640 10 monclient(hunting): authenticate will time out at 955.635385s 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f527ad9d640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f527c09eb10 0x7f527c13fb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38404/0 (socket says 172.21.15.138:38404) 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f527ad9d640 1 -- 172.21.15.138:0/2787782106 learned_addr learned my addr 172.21.15.138:0/2787782106 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5279d9b640 1 --2- 172.21.15.138:0/2787782106 >> v2:172.21.15.190:3300/0 conn(0x7f527c09e6e0 0x7f527c13d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5279d9b640 10 monclient(hunting): get_auth_request con 0x7f527c09e6e0 auth_method 0 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5279d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5279d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5279d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5279d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5279d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.194+0000 7f5279d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.198+0000 7f5279d9b640 10 monclient(hunting): handle_auth_done global_id 4231 payload 274 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.198+0000 7f5279d9b640 1 -- 172.21.15.138:0/2787782106 >> v1:172.21.15.190:6789/0 conn(0x7f527c070010 legacy=0x7f527c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.198+0000 7f5279d9b640 1 -- 172.21.15.138:0/2787782106 >> v1:172.21.15.138:6789/0 conn(0x7f527c09eb10 legacy=0x7f527c13fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.198+0000 7f5279d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.198+0000 7f5279d9b640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.198+0000 7f5279d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.198+0000 7f5279d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:07:52.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.198+0000 7f5279d9b640 1 -- 172.21.15.138:0/2787782106 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527c09cbf0 con 0x7f527c09e6e0 2024-09-11T05:07:52.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5279d9b640 1 -- 172.21.15.138:0/2787782106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c09e6e0 msgr2=0x7f527c13d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5279d9b640 1 --2- 172.21.15.138:0/2787782106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c09e6e0 0x7f527c13d640 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52700097c0 tx=0x7f527001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ca935695f40d93c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 1 -- 172.21.15.138:0/2787782106 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52700b0070 con 0x7f527c09e6e0 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: dump: 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.207 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 1 -- 172.21.15.138:0/2787782106 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52700ab600 con 0x7f527c09e6e0 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 5 monclient: authenticate success, global_id 4231 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 1 -- 172.21.15.138:0/2787782106 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52700ab9d0 con 0x7f527c09e6e0 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 4 set_mon_vals no callback set 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f527959a640 10 monclient: dump: 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 10 monclient: shutdown 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 1 -- 172.21.15.138:0/2787782106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c09e6e0 msgr2=0x7f527c13d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 1 --2- 172.21.15.138:0/2787782106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c09e6e0 0x7f527c13d640 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52700097c0 tx=0x7f527001c330 comp rx=0 tx=0).stop 2024-09-11T05:07:52.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 1 -- 172.21.15.138:0/2787782106 shutdown_connections 2024-09-11T05:07:52.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 1 --2- 172.21.15.138:0/2787782106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c09e6e0 0x7f527c13d640 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 1 -- 172.21.15.138:0/2787782106 >> 172.21.15.138:0/2787782106 conn(0x7f527c0ab830 msgr2=0x7f527c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 1 -- 172.21.15.138:0/2787782106 shutdown_connections 2024-09-11T05:07:52.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.202+0000 7f5280fbe640 1 -- 172.21.15.138:0/2787782106 wait complete. 2024-09-11T05:07:52.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient: monmap: 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 librados: starting msgr at 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 librados: starting objecter 2024-09-11T05:07:52.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 Processor -- start 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 -- start start 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 librados: setting wanted keys 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 librados: calling monclient init 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient: init 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f527c070010 0x7f527c140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient(hunting): picked mon.noname-c con 0x7f527c070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f527c09eb10 0x7f527c0a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient(hunting): picked mon.noname-a con 0x7f527c09eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c0a3680 0x7f527c0a3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient(hunting): picked mon.noname-b con 0x7f527c0a3680 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f527c143e60 con 0x7f527c09eb10 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f527c143ce0 con 0x7f527c0a3680 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f527c143fe0 con 0x7f527c070010 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5280fbe640 10 monclient(hunting): authenticate will time out at 955.645504s 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f527c070010 0x7f527c140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f527c070010 0x7f527c140a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58782/0 (socket says 172.21.15.138:58782) 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 -- 172.21.15.138:0/170936349 learned_addr learned my addr 172.21.15.138:0/170936349 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient(hunting): get_auth_request con 0x7f527c070010 auth_method 0 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f527c09eb10 0x7f527c0a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 10 monclient(hunting): get_auth_request con 0x7f527c09eb10 auth_method 0 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-09-11T05:07:52.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 -- 172.21.15.138:0/170936349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c0a3680 msgr2=0x7f527c0a3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527ad9d640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c0a3680 0x7f527c0a3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c0a3680 0x7f527c0a3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 -- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f527c09eb10 msgr2=0x7f527c0a3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f527c09eb10 0x7f527c0a3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f527a59c640 1 -- 172.21.15.138:0/170936349 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527c0724c0 con 0x7f527c070010 2024-09-11T05:07:52.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.206+0000 7f5279d9b640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f527c09eb10 0x7f527c0a3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:52.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f527a59c640 1 -- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f527c070010 msgr2=0x7f527c140a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f527a59c640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f527c070010 0x7f527c140a30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c480 tx=0x7f526c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 client.?.objecter ms_handle_connect 0x7f527c070010 2024-09-11T05:07:52.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 1 -- 172.21.15.138:0/170936349 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c012a40 con 0x7f527c070010 2024-09-11T05:07:52.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: dump: 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 1 -- 172.21.15.138:0/170936349 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526c017070 con 0x7f527c070010 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 5 monclient: authenticate success, global_id 4250 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 10 monclient: _renew_subs 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f527c1336f0 con 0x7f527c070010 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 10 monclient: _renew_subs 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f527c0a2330 con 0x7f527c070010 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 1 -- 172.21.15.138:0/170936349 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c012e40 con 0x7f527c070010 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 1 librados: init done 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 monclient: dump: 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5278d99640 4 set_mon_vals no callback set 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5260005090 con 0x7f527c070010 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 1 -- 172.21.15.138:0/170936349 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f526c01c460 con 0x7f527c070010 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 1 --2- 172.21.15.138:0/170936349 >> v2:172.21.15.190:6800/1562782189 conn(0x7f525c03cef0 0x7f525c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 1 -- 172.21.15.138:0/170936349 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f526c04efb0 con 0x7f527c070010 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 10 client.4250.objecter ms_dispatch 0x7f527c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 3 client.4250.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 3 client.4250.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f526affd640 20 client.4250.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5279d9b640 1 --2- 172.21.15.138:0/170936349 >> v2:172.21.15.190:6800/1562782189 conn(0x7f525c03cef0 0x7f525c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.210+0000 7f5279d9b640 10 monclient: get_auth_request con 0x7f525c03cef0 auth_method 0 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.214+0000 7f5279d9b640 1 --2- 172.21.15.138:0/170936349 >> v2:172.21.15.190:6800/1562782189 conn(0x7f525c03cef0 0x7f525c03f3b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f527c135200 tx=0x7f52700a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.214+0000 7f526affd640 10 client.4250.objecter ms_handle_connect 0x7f525c03cef0 2024-09-11T05:07:52.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.218+0000 7f526affd640 1 -- 172.21.15.138:0/170936349 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f526c010cf0 con 0x7f527c070010 2024-09-11T05:07:52.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.218+0000 7f526affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.218+0000 7f526affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient: monmap: 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.301550+0000 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.301550+0000 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.300 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.301 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.301 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.301 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.301 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.301 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 1 Processor -- start 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 1 -- start start 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient: init 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f75b0070010 0x7f75b0135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient(hunting): picked mon.noname-c con 0x7f75b0070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b009e6e0 0x7f75b009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient(hunting): picked mon.noname-b con 0x7f75b009e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75b01382c0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b00a8990 con 0x7f75b01382c0 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f75b00703f0 con 0x7f75b0070010 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f75b009cbf0 con 0x7f75b009e6e0 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75b4d15640 10 monclient(hunting): authenticate will time out at 955.738007s 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b009e6e0 0x7f75b009eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b009e6e0 0x7f75b009eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58798/0 (socket says 172.21.15.138:58798) 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 1 -- 172.21.15.138:0/1004021219 learned_addr learned my addr 172.21.15.138:0/1004021219 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient(hunting): get_auth_request con 0x7f75b009e6e0 auth_method 0 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75ae59c640 1 --2- 172.21.15.138:0/1004021219 >> v2:172.21.15.190:3300/0 conn(0x7f75b0070010 0x7f75b0135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75ae59c640 10 monclient(hunting): get_auth_request con 0x7f75b0070010 auth_method 0 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-11T05:07:52.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 1 -- 172.21.15.138:0/1004021219 >> v2:172.21.15.190:3300/0 conn(0x7f75b0070010 msgr2=0x7f75b0135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 1 --2- 172.21.15.138:0/1004021219 >> v2:172.21.15.190:3300/0 conn(0x7f75b0070010 0x7f75b0135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 1 -- 172.21.15.138:0/1004021219 >> v1:172.21.15.190:6789/0 conn(0x7f75b01382c0 legacy=0x7f75b0138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 1 monclient: found mon.noname-b 2024-09-11T05:07:52.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.298+0000 7f75add9b640 1 -- 172.21.15.138:0/1004021219 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b009ced0 con 0x7f75b009e6e0 2024-09-11T05:07:52.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.302+0000 7f75add9b640 1 -- 172.21.15.138:0/1004021219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b009e6e0 msgr2=0x7f75b009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.302+0000 7f75add9b640 1 --2- 172.21.15.138:0/1004021219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b009e6e0 0x7f75b009eae0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009870 tx=0x7f75a401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1bb5990bf00e102 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 1 -- 172.21.15.138:0/1004021219 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a40b2070 con 0x7f75b009e6e0 2024-09-11T05:07:52.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:52.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: dump: 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 1 -- 172.21.15.138:0/1004021219 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a40ad640 con 0x7f75b009e6e0 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 5 monclient: authenticate success, global_id 4253 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 1 -- 172.21.15.138:0/1004021219 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a40ada10 con 0x7f75b009e6e0 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75ad59a640 10 monclient: dump: 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.309 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.310 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.310 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.310 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 4 set_mon_vals no callback set 2024-09-11T05:07:52.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 10 monclient: shutdown 2024-09-11T05:07:52.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 -- 172.21.15.138:0/1004021219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b009e6e0 msgr2=0x7f75b009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 --2- 172.21.15.138:0/1004021219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b009e6e0 0x7f75b009eae0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009870 tx=0x7f75a401c3f0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 -- 172.21.15.138:0/1004021219 shutdown_connections 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 --2- 172.21.15.138:0/1004021219 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b009e6e0 0x7f75b009eae0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 --2- 172.21.15.138:0/1004021219 >> v2:172.21.15.190:3300/0 conn(0x7f75b0070010 0x7f75b0135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 -- 172.21.15.138:0/1004021219 >> 172.21.15.138:0/1004021219 conn(0x7f75b00ab830 msgr2=0x7f75b00a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 -- 172.21.15.138:0/1004021219 shutdown_connections 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 -- 172.21.15.138:0/1004021219 wait complete. 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 10 monclient: monmap: 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.312 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 librados: starting msgr at 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 librados: starting objecter 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.306+0000 7f75b4d15640 1 Processor -- start 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 -- start start 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 librados: setting wanted keys 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 librados: calling monclient init 2024-09-11T05:07:52.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient: init 2024-09-11T05:07:52.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:52.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75b0070010 0x7f75b013f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient(hunting): picked mon.noname-b con 0x7f75b0070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75b009e6e0 0x7f75b013fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient(hunting): picked mon.noname-a con 0x7f75b009e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b01382c0 0x7f75b00a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient: monmap: 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.315064+0000 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.315064+0000 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.316 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.317 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.317 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75b009e6e0 0x7f75b013fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75ae59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75b0070010 0x7f75b013f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75b009e6e0 0x7f75b013fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46320/0 (socket says 172.21.15.138:46320) 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75ae59c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75b0070010 0x7f75b013f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.138:36084/0 (socket says 172.21.15.138:36084) 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b01382c0 0x7f75b00a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b01382c0 0x7f75b00a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58802/0 (socket says 172.21.15.138:58802) 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient(hunting): picked mon.noname-c con 0x7f75b01382c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 -- 172.21.15.138:0/2047035364 learned_addr learned my addr 172.21.15.138:0/2047035364 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 -- 172.21.15.138:0/2047035364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f75b01432f0 con 0x7f75b009e6e0 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 -- 172.21.15.138:0/2047035364 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0143170 con 0x7f75b0070010 2024-09-11T05:07:52.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 1 -- 172.21.15.138:0/2047035364 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f75b0143470 con 0x7f75b01382c0 2024-09-11T05:07:52.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75b4d15640 10 monclient(hunting): authenticate will time out at 955.750273s 2024-09-11T05:07:52.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 1 Processor -- start 2024-09-11T05:07:52.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient(hunting): get_auth_request con 0x7f75b009e6e0 auth_method 0 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75ae59c640 10 monclient(hunting): get_auth_request con 0x7f75b0070010 auth_method 0 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 10 monclient(hunting): get_auth_request con 0x7f75b01382c0 auth_method 0 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 1 -- start start 2024-09-11T05:07:52.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient: init 2024-09-11T05:07:52.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient: monmap: 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.322 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.315995+0000 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.315995+0000 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 -- 172.21.15.138:0/2047035364 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b01382c0 msgr2=0x7f75b00a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 --2- 172.21.15.138:0/2047035364 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b01382c0 0x7f75b00a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 -- 172.21.15.138:0/2047035364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75b0070010 msgr2=0x7f75b013f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 --2- 172.21.15.138:0/2047035364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75b0070010 0x7f75b013f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 monclient: found mon.noname-a 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faa94070010 0x7faa94135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7f75add9b640 1 -- 172.21.15.138:0/2047035364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b00a2330 con 0x7f75b009e6e0 2024-09-11T05:07:52.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient(hunting): picked mon.noname-c con 0x7faa94070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:52.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7faa9409e6e0 0x7faa941382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient(hunting): picked mon.noname-a con 0x7faa9409e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:52.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 1 Processor -- start 2024-09-11T05:07:52.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa9409eb10 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:52.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 1 -- start start 2024-09-11T05:07:52.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient: init 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7faa9409e6e0 0x7faa941382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa93fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7faa9409eb10 0x7faa94138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38406/0 (socket says 172.21.15.138:38406) 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa93fff640 1 -- 172.21.15.138:0/3574943362 learned_addr learned my addr 172.21.15.138:0/3574943362 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 1 -- 172.21.15.138:0/3574943362 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa940a8990 con 0x7faa9409eb10 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 1 -- 172.21.15.138:0/3574943362 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7faa940703f0 con 0x7faa9409e6e0 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 1 -- 172.21.15.138:0/3574943362 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7faa9409cbf0 con 0x7faa94070010 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa99f5c640 10 monclient(hunting): authenticate will time out at 955.751380s 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient(hunting): get_auth_request con 0x7faa9409e6e0 auth_method 0 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa937fe640 1 --2- 172.21.15.138:0/3574943362 >> v2:172.21.15.190:3300/0 conn(0x7faa94070010 0x7faa94135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa937fe640 10 monclient(hunting): get_auth_request con 0x7faa94070010 auth_method 0 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 1 -- 172.21.15.138:0/3574943362 >> v2:172.21.15.190:3300/0 conn(0x7faa94070010 msgr2=0x7faa94135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 1 --2- 172.21.15.138:0/3574943362 >> v2:172.21.15.190:3300/0 conn(0x7faa94070010 0x7faa94135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 1 -- 172.21.15.138:0/3574943362 >> v1:172.21.15.138:6789/0 conn(0x7faa9409eb10 legacy=0x7faa94138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 1 monclient: found mon.noname-a 2024-09-11T05:07:52.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7faa92ffd640 1 -- 172.21.15.138:0/3574943362 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9409ced0 con 0x7faa9409e6e0 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbcd4070010 0x7fbcd4135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd4070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbcd409e6e0 0x7fbcd41382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd409e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbcd409eb10 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.310+0000 7fbcd9583640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd40a8990 con 0x7fbcd409eb10 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd9583640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd9583640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd40703f0 con 0x7fbcd409e6e0 2024-09-11T05:07:52.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd9583640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd9583640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd409cbf0 con 0x7fbcd4070010 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd9583640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd9583640 10 monclient(hunting): authenticate will time out at 955.752146s 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbcd4070010 0x7fbcd4135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd37fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fbcd409eb10 0x7fbcd4138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.138:55152/0 (socket says 172.21.15.138:55152) 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd37fe640 1 -- 172.21.15.138:0/1928996930 learned_addr learned my addr 172.21.15.138:0/1928996930 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient(hunting): get_auth_request con 0x7fbcd4070010 auth_method 0 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 1 -- 172.21.15.138:0/1928996930 >> v2:172.21.15.138:3300/0 conn(0x7fbcd409e6e0 msgr2=0x7fbcd41382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 1 --2- 172.21.15.138:0/1928996930 >> v2:172.21.15.138:3300/0 conn(0x7fbcd409e6e0 0x7fbcd41382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 1 -- 172.21.15.138:0/1928996930 >> v1:172.21.15.190:6789/0 conn(0x7fbcd409eb10 legacy=0x7fbcd4138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:07:52.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7fbcd2ffd640 1 -- 172.21.15.138:0/1928996930 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd409ced0 con 0x7fbcd4070010 2024-09-11T05:07:52.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75add9b640 1 -- 172.21.15.138:0/2047035364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75b009e6e0 msgr2=0x7f75b013fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75add9b640 1 --2- 172.21.15.138:0/2047035364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75b009e6e0 0x7f75b013fc60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f75a401aaa0 tx=0x7f75a4019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 client.?.objecter ms_handle_connect 0x7f75b009e6e0 2024-09-11T05:07:52.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 1 -- 172.21.15.138:0/2047035364 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a40b2040 con 0x7f75b009e6e0 2024-09-11T05:07:52.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: dump: 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 1 -- 172.21.15.138:0/2047035364 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a40bb070 con 0x7f75b009e6e0 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 5 monclient: authenticate success, global_id 4254 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 10 monclient: _renew_subs 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 1 -- 172.21.15.138:0/2047035364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b01401a0 con 0x7f75b009e6e0 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 1 -- 172.21.15.138:0/2047035364 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a40ade60 con 0x7f75b009e6e0 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759effd640 10 monclient: dump: 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.337 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f759ffff640 4 set_mon_vals no callback set 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 10 monclient: _renew_subs 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 1 -- 172.21.15.138:0/2047035364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b013ac00 con 0x7f75b009e6e0 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.314+0000 7f75b4d15640 1 librados: init done 2024-09-11T05:07:52.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 1 -- 172.21.15.138:0/3574943362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 msgr2=0x7faa941382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 1 --2- 172.21.15.138:0/3574943362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa941382c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faa9406daa0 tx=0x7faa8801c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4582b9f9fd2f6f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 1 -- 172.21.15.138:0/3574943362 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880b0070 con 0x7faa9409e6e0 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: dump: 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.339 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 5 monclient: authenticate success, global_id 4257 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 1 -- 172.21.15.138:0/3574943362 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa880ab600 con 0x7faa9409e6e0 2024-09-11T05:07:52.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 1 -- 172.21.15.138:0/3574943362 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880ab9d0 con 0x7faa9409e6e0 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 4 set_mon_vals no callback set 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa927fc640 10 monclient: dump: 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.341 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.342 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.342 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759cff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759cff9640 1 -- 172.21.15.138:0/2047035364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7588005090 con 0x7f75b009e6e0 2024-09-11T05:07:52.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759effd640 1 -- 172.21.15.138:0/2047035364 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f75a40b5810 con 0x7f75b009e6e0 2024-09-11T05:07:52.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient: shutdown 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- 172.21.15.138:0/3574943362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 msgr2=0x7faa941382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 --2- 172.21.15.138:0/3574943362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa941382c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faa9406daa0 tx=0x7faa8801c310 comp rx=0 tx=0).stop 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- 172.21.15.138:0/3574943362 shutdown_connections 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 --2- 172.21.15.138:0/3574943362 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa941382c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 --2- 172.21.15.138:0/3574943362 >> v2:172.21.15.190:3300/0 conn(0x7faa94070010 0x7faa94135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- 172.21.15.138:0/3574943362 >> 172.21.15.138:0/3574943362 conn(0x7faa940ab830 msgr2=0x7faa940a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759effd640 1 --2- 172.21.15.138:0/2047035364 >> v2:172.21.15.190:6800/1562782189 conn(0x7f759003cef0 0x7f759003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759effd640 1 -- 172.21.15.138:0/2047035364 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f75a40e91a0 con 0x7f75b009e6e0 2024-09-11T05:07:52.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759effd640 10 client.4254.objecter ms_dispatch 0x7f75b0000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759effd640 3 client.4254.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759effd640 3 client.4254.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- 172.21.15.138:0/3574943362 shutdown_connections 2024-09-11T05:07:52.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- 172.21.15.138:0/3574943362 wait complete. 2024-09-11T05:07:52.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f759effd640 20 client.4254.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f75ae59c640 1 --2- 172.21.15.138:0/2047035364 >> v2:172.21.15.190:6800/1562782189 conn(0x7f759003cef0 0x7f759003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7f75ae59c640 10 monclient: get_auth_request con 0x7f759003cef0 auth_method 0 2024-09-11T05:07:52.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient: monmap: 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 librados: starting msgr at 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 librados: starting objecter 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 Processor -- start 2024-09-11T05:07:52.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- start start 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 librados: setting wanted keys 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 librados: calling monclient init 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient: init 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa94070010 0x7faa94131eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient(hunting): picked mon.noname-b con 0x7faa94070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa940a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa940a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa940a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46354/0 (socket says 172.21.15.138:46354) 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 1 -- 172.21.15.138:0/632500069 learned_addr learned my addr 172.21.15.138:0/632500069 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa937fe640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa94070010 0x7faa94131eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient(hunting): picked mon.noname-a con 0x7faa9409e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa9409eb10 0x7faa940a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient(hunting): picked mon.noname-c con 0x7faa9409eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7faa94142e80 con 0x7faa9409e6e0 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa94139120 con 0x7faa94070010 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7faa94143000 con 0x7faa9409eb10 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa99f5c640 10 monclient(hunting): authenticate will time out at 955.759640s 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 10 monclient(hunting): get_auth_request con 0x7faa9409e6e0 auth_method 0 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa937fe640 10 monclient(hunting): get_auth_request con 0x7faa94070010 auth_method 0 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa93fff640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa9409eb10 0x7faa940a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa93fff640 10 monclient(hunting): get_auth_request con 0x7faa9409eb10 auth_method 0 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa93fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.318+0000 7faa937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 1 -- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa9409eb10 msgr2=0x7faa940a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa9409eb10 0x7faa940a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 1 -- 172.21.15.138:0/632500069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa94070010 msgr2=0x7faa94131eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa94070010 0x7faa94131eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 1 monclient: found mon.noname-a 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 1 -- 172.21.15.138:0/632500069 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa940aa8c0 con 0x7faa9409e6e0 2024-09-11T05:07:52.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 1 -- 172.21.15.138:0/1928996930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd4070010 msgr2=0x7fbcd4135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 1 --2- 172.21.15.138:0/1928996930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd4070010 0x7fbcd4135450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbcd406daa0 tx=0x7fbcc401c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbe41b04a9c67e6e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7f759effd640 10 client.4254.objecter ms_handle_connect 0x7f759003cef0 2024-09-11T05:07:52.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7f75ae59c640 1 --2- 172.21.15.138:0/2047035364 >> v2:172.21.15.190:6800/1562782189 conn(0x7f759003cef0 0x7f759003f3b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009af0 tx=0x7f75a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 1 -- 172.21.15.138:0/1928996930 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc40b0070 con 0x7fbcd4070010 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: dump: 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.353 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 5 monclient: authenticate success, global_id 4243 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 1 -- 172.21.15.138:0/1928996930 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc40ab600 con 0x7fbcd4070010 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 4 set_mon_vals no callback set 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 1 -- 172.21.15.138:0/1928996930 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc40ab9d0 con 0x7fbcd4070010 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd1ffb640 10 monclient: dump: 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.355 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient: shutdown 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- 172.21.15.138:0/1928996930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd4070010 msgr2=0x7fbcd4135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 --2- 172.21.15.138:0/1928996930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd4070010 0x7fbcd4135450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbcd406daa0 tx=0x7fbcc401c310 comp rx=0 tx=0).stop 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- 172.21.15.138:0/1928996930 shutdown_connections 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 --2- 172.21.15.138:0/1928996930 >> v2:172.21.15.138:3300/0 conn(0x7fbcd409e6e0 0x7fbcd41382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 --2- 172.21.15.138:0/1928996930 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd4070010 0x7fbcd4135450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- 172.21.15.138:0/1928996930 >> 172.21.15.138:0/1928996930 conn(0x7fbcd40ab830 msgr2=0x7fbcd40a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- 172.21.15.138:0/1928996930 shutdown_connections 2024-09-11T05:07:52.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- 172.21.15.138:0/1928996930 wait complete. 2024-09-11T05:07:52.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 1 -- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 msgr2=0x7faa940a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa92ffd640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa940a01b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa8801a9c0 tx=0x7faa880190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa7bfff640 10 client.?.objecter ms_handle_connect 0x7faa9409e6e0 2024-09-11T05:07:52.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7faa7bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient: monmap: 2024-09-11T05:07:52.358 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 librados: starting msgr at 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 librados: starting objecter 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 Processor -- start 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- start start 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 librados: setting wanted keys 2024-09-11T05:07:52.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 librados: calling monclient init 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient: init 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbcd4070010 0x7fbcd40a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd4070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbcd409e6e0 0x7fbcd40a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd409e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd409eb10 0x7fbcd40a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient(hunting): picked mon.noname-b con 0x7fbcd409eb10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd4142e80 con 0x7fbcd409e6e0 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd4139120 con 0x7fbcd409eb10 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd4143000 con 0x7fbcd4070010 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd9583640 10 monclient(hunting): authenticate will time out at 955.763669s 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbcd409e6e0 0x7fbcd40a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbcd409e6e0 0x7fbcd40a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46366/0 (socket says 172.21.15.138:46366) 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 1 -- 172.21.15.138:0/2732131836 learned_addr learned my addr 172.21.15.138:0/2732131836 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 10 monclient(hunting): get_auth_request con 0x7fbcd409e6e0 auth_method 0 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbcd4070010 0x7fbcd40a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 10 monclient(hunting): get_auth_request con 0x7fbcd4070010 auth_method 0 2024-09-11T05:07:52.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.322+0000 7fbcd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd37fe640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd409eb10 0x7fbcd40a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 1 -- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbcd4070010 msgr2=0x7fbcd40a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbcd4070010 0x7fbcd40a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 1 -- 172.21.15.138:0/2732131836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd409eb10 msgr2=0x7fbcd40a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd409eb10 0x7fbcd40a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 1 monclient: found mon.noname-a 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 1 -- 172.21.15.138:0/2732131836 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd413add0 con 0x7fbcd409e6e0 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd37fe640 10 monclient: get_auth_request con 0x7fbcd409eb10 auth_method 0 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd37fe640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd409eb10 0x7fbcd40a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7f759effd640 1 -- 172.21.15.138:0/2047035364 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f75a4016130 con 0x7f75b009e6e0 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7f759effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7f759effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:52.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 1 -- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbcd409e6e0 msgr2=0x7fbcd40a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcd27fc640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbcd409e6e0 0x7fbcd40a0ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800b580 tx=0x7fbcc800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcbb7fe640 10 client.?.objecter ms_handle_connect 0x7fbcd409e6e0 2024-09-11T05:07:52.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.326+0000 7fbcbb7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 1 -- 172.21.15.138:0/632500069 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880b9020 con 0x7faa9409e6e0 2024-09-11T05:07:52.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: dump: 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 5 monclient: authenticate success, global_id 4260 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 10 monclient: _renew_subs 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa940a2330 con 0x7faa9409e6e0 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 1 -- 172.21.15.138:0/2732131836 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8014070 con 0x7fbcd409e6e0 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: dump: 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.366 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.367 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 10 monclient: _renew_subs 2024-09-11T05:07:52.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa9413acb0 con 0x7faa9409e6e0 2024-09-11T05:07:52.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 1 librados: init done 2024-09-11T05:07:52.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 1 -- 172.21.15.138:0/632500069 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa880b58a0 con 0x7faa9409e6e0 2024-09-11T05:07:52.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 1 -- 172.21.15.138:0/632500069 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880b0040 con 0x7faa9409e6e0 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 monclient: dump: 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.369 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa91ffb640 4 set_mon_vals no callback set 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 1 -- 172.21.15.138:0/632500069 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7faa880b43e0 con 0x7faa9409e6e0 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa74005090 con 0x7faa9409e6e0 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 1 --2- 172.21.15.138:0/632500069 >> v2:172.21.15.190:6800/1562782189 conn(0x7faa8003cf00 0x7faa8003f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 1 -- 172.21.15.138:0/632500069 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7faa880e9520 con 0x7faa9409e6e0 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 10 client.4260.objecter ms_dispatch 0x7faa94000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 3 client.4260.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 3 client.4260.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa7bfff640 20 client.4260.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 1 -- 172.21.15.138:0/2732131836 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc8010aa0 con 0x7fbcd409e6e0 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 1 -- 172.21.15.138:0/2732131836 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc800f3c0 con 0x7fbcd409e6e0 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcbb7fe640 10 monclient: dump: 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.371 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.372 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.372 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.372 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.372 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.372 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd17fa640 4 set_mon_vals no callback set 2024-09-11T05:07:52.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa937fe640 1 --2- 172.21.15.138:0/632500069 >> v2:172.21.15.190:6800/1562782189 conn(0x7faa8003cf00 0x7faa8003f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7faa937fe640 10 monclient: get_auth_request con 0x7faa8003cf00 auth_method 0 2024-09-11T05:07:52.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 5 monclient: authenticate success, global_id 4263 2024-09-11T05:07:52.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 10 monclient: _renew_subs 2024-09-11T05:07:52.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd4139910 con 0x7fbcd409e6e0 2024-09-11T05:07:52.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 10 monclient: _renew_subs 2024-09-11T05:07:52.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd41330b0 con 0x7fbcd409e6e0 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 1 librados: init done 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.330+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd4135450 con 0x7fbcd409e6e0 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.334+0000 7faa937fe640 1 --2- 172.21.15.138:0/632500069 >> v2:172.21.15.190:6800/1562782189 conn(0x7faa8003cf00 0x7faa8003f3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faa840079c0 tx=0x7faa84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.334+0000 7faa7bfff640 10 client.4260.objecter ms_handle_connect 0x7faa8003cf00 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.334+0000 7faa7bfff640 1 -- 172.21.15.138:0/632500069 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7faa880c3070 con 0x7faa9409e6e0 2024-09-11T05:07:52.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.334+0000 7faa7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.334+0000 7faa7bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:52.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 1 -- 172.21.15.138:0/2732131836 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fbcc800f560 con 0x7fbcd409e6e0 2024-09-11T05:07:52.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 1 --2- 172.21.15.138:0/2732131836 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbcc003ced0 0x7fbcc003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 1 -- 172.21.15.138:0/2732131836 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fbcc804f8f0 con 0x7fbcd409e6e0 2024-09-11T05:07:52.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 10 client.4263.objecter ms_dispatch 0x7fbcd4000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 3 client.4263.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 3 client.4263.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 20 client.4263.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 1 -- 172.21.15.138:0/2732131836 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbcc800dc60 con 0x7fbcd409e6e0 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcbb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcd2ffd640 1 --2- 172.21.15.138:0/2732131836 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbcc003ced0 0x7fbcc003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.342+0000 7fbcd2ffd640 10 monclient: get_auth_request con 0x7fbcc003ced0 auth_method 0 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.346+0000 7fbcd2ffd640 1 --2- 172.21.15.138:0/2732131836 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbcc003ced0 0x7fbcc003f390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4135200 tx=0x7fbcc40a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.346+0000 7fbcbb7fe640 10 client.4263.objecter ms_handle_connect 0x7fbcc003ced0 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient: monmap: 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.358637+0000 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.358637+0000 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.378 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 1 Processor -- start 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 1 -- start start 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient: init 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0258070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f025809e6e0 0x7f025813d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient(hunting): picked mon.noname-a con 0x7f025809e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f025809eb10 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:52.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0258135b60 con 0x7f025809eb10 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02580a8990 con 0x7f0258070010 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f02580703f0 con 0x7f025809e6e0 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f025d71e640 10 monclient(hunting): authenticate will time out at 955.794911s 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f02577fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f025809eb10 0x7f025813fb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38408/0 (socket says 172.21.15.138:38408) 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f02577fe640 1 -- 172.21.15.138:0/3575874872 learned_addr learned my addr 172.21.15.138:0/3575874872 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 1 --2- 172.21.15.138:0/3575874872 >> v2:172.21.15.138:3300/0 conn(0x7f025809e6e0 0x7f025813d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 10 monclient(hunting): get_auth_request con 0x7f025809e6e0 auth_method 0 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 1 -- 172.21.15.138:0/3575874872 >> v2:172.21.15.138:3300/0 conn(0x7f025809e6e0 msgr2=0x7f025813d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:07:52.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 1 -- 172.21.15.138:0/3575874872 >> v2:172.21.15.138:3300/0 conn(0x7f025809e6e0 msgr2=0x7f025813d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 1 --2- 172.21.15.138:0/3575874872 >> v2:172.21.15.138:3300/0 conn(0x7f025809e6e0 0x7f025813d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.354+0000 7f024ffff640 1 --2- 172.21.15.138:0/3575874872 >> v2:172.21.15.138:3300/0 conn(0x7f025809e6e0 0x7f025813d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02577fe640 1 -- 172.21.15.138:0/3575874872 >> v1:172.21.15.138:6789/0 conn(0x7f025809eb10 legacy=0x7f025813fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 1 -- 172.21.15.138:0/3575874872 >> v1:172.21.15.190:6789/0 conn(0x7f0258070010 legacy=0x7f0258135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3329632125 0 0) 0x7f0258135b60 con 0x7f025809eb10 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): _negotiate 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): my global_id is 4266 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0244003530 con 0x7f025809eb10 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2160837392 0 0) 0x7f0244003530 con 0x7f025809eb10 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 >> v2:172.21.15.138:3300/0 conn(0x7f025809e6e0 msgr2=0x7f025813d640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 --2- 172.21.15.138:0/3575874872 >> v2:172.21.15.138:3300/0 conn(0x7f025809e6e0 0x7f025813d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 >> v1:172.21.15.190:6789/0 conn(0x7f0258070010 legacy=0x7f0258135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025809cbf0 con 0x7f025809eb10 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f0250001df0 con 0x7f025809eb10 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f025d71e640 5 monclient: authenticate success, global_id 4266 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f025d71e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: dump: 2024-09-11T05:07:52.382 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 >> v1:172.21.15.138:6789/0 conn(0x7f025809eb10 legacy=0x7f025813fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244003a40 0x7f0244003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): picked mon.b con 0x7f0244003a40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0244004810 0x7f0244004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient: get_monmap_and_config 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient: monmap: 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:52.363875+0000 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:52.363875+0000 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.384 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0244004810 0x7f0244004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): picked mon.a con 0x7f0244004810 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244003a40 0x7f0244003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02440053d0 0x7f024400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): picked mon.c con 0x7f02440053d0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f02580703f0 con 0x7f0244004810 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f024400c240 con 0x7f0244003a40 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f024400c440 con 0x7f02440053d0 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02567fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient(hunting): get_auth_request con 0x7f0244004810 auth_method 0 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 10 monclient(hunting): get_auth_request con 0x7f0244003a40 auth_method 0 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f025d71e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f0256ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient(hunting): handle_auth_done global_id 4266 payload 293 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 1 -- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02440053d0 msgr2=0x7f024400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02577fe640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02440053d0 0x7f024400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02440053d0 0x7f024400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 1 -- 172.21.15.138:0/3575874872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244003a40 msgr2=0x7f0244003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244003a40 0x7f0244003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 1 monclient: found mon.a 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 1 -- 172.21.15.138:0/3575874872 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025809cbf0 con 0x7f0244004810 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f024ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02577fe640 10 monclient: get_auth_request con 0x7f02440053d0 auth_method 0 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02577fe640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02440053d0 0x7f024400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:52.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f02577fe640 1 -- 172.21.15.138:0/3575874872 reap_dead start 2024-09-11T05:07:52.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 1 Processor -- start 2024-09-11T05:07:52.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 1 -- start start 2024-09-11T05:07:52.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient: init 2024-09-11T05:07:52.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-11T05:07:52.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31f00a4aa0 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f013b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient(hunting): picked mon.noname-b con 0x7f31f00a5590 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f31f013bdc0 0x7f31f013e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient(hunting): picked mon.noname-c con 0x7f31f013bdc0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f00a1660 con 0x7f31f00a4aa0 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f31f00a0cc0 con 0x7f31f013bdc0 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f31f00a1d60 con 0x7f31f00a5590 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f6fd3640 10 monclient(hunting): authenticate will time out at 955.799657s 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f4d6f640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f31f00a4aa0 0x7f31f00a4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38414/0 (socket says 172.21.15.138:38414) 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31f4d6f640 1 -- 172.21.15.138:0/337785121 learned_addr learned my addr 172.21.15.138:0/337785121 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:52.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31effff640 1 --2- 172.21.15.138:0/337785121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f013b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31effff640 10 monclient(hunting): get_auth_request con 0x7f31f00a5590 auth_method 0 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.358+0000 7f31effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31f5570640 1 --2- 172.21.15.138:0/337785121 >> v2:172.21.15.190:3300/0 conn(0x7f31f013bdc0 0x7f31f013e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31f5570640 10 monclient(hunting): get_auth_request con 0x7f31f013bdc0 auth_method 0 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31f5570640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31f5570640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31f5570640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-09-11T05:07:52.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 1 -- 172.21.15.138:0/337785121 >> v2:172.21.15.190:3300/0 conn(0x7f31f013bdc0 msgr2=0x7f31f013e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 1 --2- 172.21.15.138:0/337785121 >> v2:172.21.15.190:3300/0 conn(0x7f31f013bdc0 0x7f31f013e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 1 -- 172.21.15.138:0/337785121 >> v1:172.21.15.138:6789/0 conn(0x7f31f00a4aa0 legacy=0x7f31f00a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:52.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 1 monclient: found mon.noname-b 2024-09-11T05:07:52.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.362+0000 7f31effff640 1 -- 172.21.15.138:0/337785121 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f00a2040 con 0x7f31f00a5590 2024-09-11T05:07:52.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f024ffff640 1 -- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0244004810 msgr2=0x7f0244004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f024ffff640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0244004810 0x7f0244004c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f024000c820 tx=0x7f024001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fcc70a88ec0d730 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02400b1070 con 0x7f0244004810 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 10 monclient: dump: 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.394 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02400ac630 con 0x7f0244004810 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 1 -- 172.21.15.138:0/3575874872 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02400a8050 con 0x7f0244004810 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 4 set_mon_vals no callback set 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f02567fc640 10 monclient: dump: 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.395 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.396 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.396 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.396 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.396 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.396 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.396 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31effff640 1 -- 172.21.15.138:0/337785121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 msgr2=0x7f31f013b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31effff640 1 --2- 172.21.15.138:0/337785121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f013b880 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f31e40097c0 tx=0x7f31e401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e0a26fb20e3c0bb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient: shutdown 2024-09-11T05:07:52.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0244004810 msgr2=0x7f0244004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0244004810 0x7f0244004c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f024000c820 tx=0x7f024001c4c0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- 172.21.15.138:0/3575874872 shutdown_connections 2024-09-11T05:07:52.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 --2- 172.21.15.138:0/3575874872 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0244004810 0x7f0244004c90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- 172.21.15.138:0/3575874872 >> 172.21.15.138:0/3575874872 conn(0x7f02580ab830 msgr2=0x7f02580a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 1 -- 172.21.15.138:0/337785121 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b0070 con 0x7f31f00a5590 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: dump: 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 1 -- 172.21.15.138:0/337785121 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e40ab600 con 0x7f31f00a5590 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 5 monclient: authenticate success, global_id 4265 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 1 -- 172.21.15.138:0/337785121 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40ab9d0 con 0x7f31f00a5590 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31ef7fe640 10 monclient: dump: 2024-09-11T05:07:52.399 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 4 set_mon_vals no callback set 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- 172.21.15.138:0/3575874872 shutdown_connections 2024-09-11T05:07:52.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- 172.21.15.138:0/3575874872 wait complete. 2024-09-11T05:07:52.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient: shutdown 2024-09-11T05:07:52.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- 172.21.15.138:0/337785121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 msgr2=0x7f31f013b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/337785121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f013b880 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f31e40097c0 tx=0x7f31e401c330 comp rx=0 tx=0).stop 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- 172.21.15.138:0/337785121 shutdown_connections 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/337785121 >> v2:172.21.15.190:3300/0 conn(0x7f31f013bdc0 0x7f31f013e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/337785121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f013b880 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- 172.21.15.138:0/337785121 >> 172.21.15.138:0/337785121 conn(0x7f31f00a9690 msgr2=0x7f31f00a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- 172.21.15.138:0/337785121 shutdown_connections 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- 172.21.15.138:0/337785121 wait complete. 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient: monmap: 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.402 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 librados: starting msgr at 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 librados: starting objecter 2024-09-11T05:07:52.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient: build_initial_monmap 2024-09-11T05:07:52.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 Processor -- start 2024-09-11T05:07:52.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient: monmap: 2024-09-11T05:07:52.404 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:52.404 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:52.404 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:52.404 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:52.404 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:52.404 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 librados: starting msgr at 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 librados: starting objecter 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 Processor -- start 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- start start 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 librados: setting wanted keys 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 librados: calling monclient init 2024-09-11T05:07:52.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient: init 2024-09-11T05:07:52.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- start start 2024-09-11T05:07:52.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 librados: setting wanted keys 2024-09-11T05:07:52.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 librados: calling monclient init 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient: init 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f00a4aa0 0x7f31f0142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient(hunting): picked mon.noname-a con 0x7f31f00a4aa0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f0140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient(hunting): picked mon.noname-c con 0x7f31f00a5590 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f013bdc0 0x7f31f0141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient(hunting): picked mon.noname-b con 0x7f31f013bdc0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f31f009fd50 con 0x7f31f00a4aa0 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f31f009fbd0 con 0x7f31f013bdc0 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f31f006ec20 con 0x7f31f00a5590 2024-09-11T05:07:52.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f31f6fd3640 10 monclient(hunting): authenticate will time out at 955.807820s 2024-09-11T05:07:52.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:52.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:52.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244004810 0x7f0258121790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient(hunting): picked mon.noname-b con 0x7f0244004810 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:52.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0258121cd0 0x7f02581220d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient(hunting): picked mon.noname-a con 0x7f0258121cd0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02581d0440 0x7f02581d08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient(hunting): picked mon.noname-c con 0x7f02581d0440 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0258138c50 con 0x7f0258121cd0 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0258138ad0 con 0x7f0244004810 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0258138dd0 con 0x7f02581d0440 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.366+0000 7f025d71e640 10 monclient(hunting): authenticate will time out at 955.807834s 2024-09-11T05:07:52.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02581d0440 0x7f02581d08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f0140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f0140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58842/0 (socket says 172.21.15.138:58842) 2024-09-11T05:07:52.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 -- 172.21.15.138:0/2975221322 learned_addr learned my addr 172.21.15.138:0/2975221322 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient(hunting): get_auth_request con 0x7f31f00a5590 auth_method 0 2024-09-11T05:07:52.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f4d6f640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f00a4aa0 0x7f31f0142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02581d0440 0x7f02581d08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58840/0 (socket says 172.21.15.138:58840) 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 -- 172.21.15.138:0/1068257042 learned_addr learned my addr 172.21.15.138:0/1068257042 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient(hunting): get_auth_request con 0x7f02581d0440 auth_method 0 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024ffff640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0258121cd0 0x7f02581220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f4d6f640 10 monclient(hunting): get_auth_request con 0x7f31f00a4aa0 auth_method 0 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f4d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f4d6f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f4d6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f5570640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f013bdc0 0x7f31f0141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f5570640 10 monclient(hunting): get_auth_request con 0x7f31f013bdc0 auth_method 0 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f5570640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f5570640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f5570640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f4d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f4d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f4d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f0256ffd640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244004810 0x7f0258121790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f0256ffd640 10 monclient(hunting): get_auth_request con 0x7f0244004810 auth_method 0 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f0256ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f0256ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f0256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024ffff640 10 monclient(hunting): get_auth_request con 0x7f0258121cd0 auth_method 0 2024-09-11T05:07:52.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f5570640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f5570640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31f5570640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-09-11T05:07:52.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 -- 172.21.15.138:0/2975221322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f013bdc0 msgr2=0x7f31f0141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f013bdc0 0x7f31f0141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 -- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f00a4aa0 msgr2=0x7f31f0142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f00a4aa0 0x7f31f0142f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f0256ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f0256ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f0256ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-11T05:07:52.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 -- 172.21.15.138:0/1068257042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244004810 msgr2=0x7f0258121790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 -- 172.21.15.138:0/2975221322 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f0142210 con 0x7f31f00a5590 2024-09-11T05:07:52.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244004810 0x7f0258121790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 -- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0258121cd0 msgr2=0x7f02581220d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0258121cd0 0x7f02581220d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:52.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 monclient: found mon.noname-c 2024-09-11T05:07:52.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:52.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 -- 172.21.15.138:0/1068257042 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0258123dc0 con 0x7f02581d0440 2024-09-11T05:07:52.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 -- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 msgr2=0x7f31f0140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f31effff640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f0140f10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f31e401cbd0 tx=0x7f31e40190d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 -- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02581d0440 msgr2=0x7f02581d08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02577fe640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02581d0440 0x7f02581d08d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f02580a22e0 tx=0x7f025000b4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 client.?.objecter ms_handle_connect 0x7f02581d0440 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 1 -- 172.21.15.138:0/1068257042 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025000bd00 con 0x7f02581d0440 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: dump: 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.419 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 5 monclient: authenticate success, global_id 4268 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 10 monclient: _renew_subs 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02581240a0 con 0x7f02581d0440 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 10 monclient: _renew_subs 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02580a5480 con 0x7f02581d0440 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f025d71e640 1 librados: init done 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 1 -- 172.21.15.138:0/1068257042 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0250007110 con 0x7f02581d0440 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 1 -- 172.21.15.138:0/1068257042 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025000bea0 con 0x7f02581d0440 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f02557fa640 4 set_mon_vals no callback set 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.370+0000 7f024f7fe640 10 monclient: dump: 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.421 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f024f7fe640 1 -- 172.21.15.138:0/1068257042 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f02500074c0 con 0x7f02581d0440 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f025d71e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f025d71e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f025d71e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023c005090 con 0x7f02581d0440 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f024f7fe640 1 --2- 172.21.15.138:0/1068257042 >> v2:172.21.15.190:6800/1562782189 conn(0x7f023803cef0 0x7f023803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f024f7fe640 1 -- 172.21.15.138:0/1068257042 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f025004f910 con 0x7f02581d0440 2024-09-11T05:07:52.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f024f7fe640 10 client.4268.objecter ms_dispatch 0x7f0258000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f024f7fe640 3 client.4268.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f024f7fe640 3 client.4268.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f024f7fe640 20 client.4268.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f0256ffd640 1 --2- 172.21.15.138:0/1068257042 >> v2:172.21.15.190:6800/1562782189 conn(0x7f023803cef0 0x7f023803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.374+0000 7f0256ffd640 10 monclient: get_auth_request con 0x7f023803cef0 auth_method 0 2024-09-11T05:07:52.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f0256ffd640 1 --2- 172.21.15.138:0/1068257042 >> v2:172.21.15.190:6800/1562782189 conn(0x7f023803cef0 0x7f023803f3b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0248007ad0 tx=0x7f02480067e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f024f7fe640 10 client.4268.objecter ms_handle_connect 0x7f023803cef0 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 client.?.objecter ms_handle_connect 0x7f31f00a5590 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 1 -- 172.21.15.138:0/2975221322 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b9020 con 0x7f31f00a5590 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: dump: 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.425 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: _finish_auth 0 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 1 -- 172.21.15.138:0/2975221322 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e40b5890 con 0x7f31f00a5590 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 5 monclient: authenticate success, global_id 4271 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 10 monclient: _renew_subs 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f00a7b20 con 0x7f31f00a5590 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 10 monclient: _renew_subs 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f00a0280 con 0x7f31f00a5590 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 1 librados: init done 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 1 -- 172.21.15.138:0/2975221322 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b0040 con 0x7f31f00a5590 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31eeffd640 4 set_mon_vals no callback set 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31ed7fa640 10 monclient: dump: 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.427 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.378+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d0005090 con 0x7f31f00a5590 2024-09-11T05:07:52.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f024f7fe640 1 -- 172.21.15.138:0/1068257042 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f025004e7d0 con 0x7f02581d0440 2024-09-11T05:07:52.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f024f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f024f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f31ed7fa640 1 -- 172.21.15.138:0/2975221322 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f31e40c7430 con 0x7f31f00a5590 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f31ed7fa640 1 --2- 172.21.15.138:0/2975221322 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31dc03cf20 0x7f31dc03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f31ed7fa640 1 -- 172.21.15.138:0/2975221322 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f31e40e9a40 con 0x7f31f00a5590 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f31ed7fa640 10 client.4271.objecter ms_dispatch 0x7f31f0000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f31ed7fa640 3 client.4271.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f31ed7fa640 3 client.4271.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.382+0000 7f31ed7fa640 20 client.4271.objecter dump_active .. 0 homeless 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.386+0000 7f31f4d6f640 1 --2- 172.21.15.138:0/2975221322 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31dc03cf20 0x7f31dc03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:52.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.386+0000 7f31f4d6f640 10 monclient: get_auth_request con 0x7f31dc03cf20 auth_method 0 2024-09-11T05:07:52.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.386+0000 7f31f4d6f640 1 --2- 172.21.15.138:0/2975221322 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31dc03cf20 0x7f31dc03f3e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31e0007ca0 tx=0x7f31e0007bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:52.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.386+0000 7f31ed7fa640 10 client.4271.objecter ms_handle_connect 0x7f31dc03cf20 2024-09-11T05:07:52.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.390+0000 7f31ed7fa640 1 -- 172.21.15.138:0/2975221322 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f31e40c3030 con 0x7f31f00a5590 2024-09-11T05:07:52.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.390+0000 7f31ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:52.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.390+0000 7f31ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:52.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.462+0000 7f5280fbe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T05:07:52.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.462+0000 7f5280fbe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T05:07:52.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.462+0000 7f5280fbe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.462+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5260005380 con 0x7f527c070010 2024-09-11T05:07:52.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f526affd640 1 -- 172.21.15.138:0/170936349 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f526c01b410 con 0x7f527c070010 2024-09-11T05:07:52.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f526affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T05:07:52.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f526affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:52.468 INFO:teuthology.orchestra.run.smithi138.stdout:47244640259 2024-09-11T05:07:52.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 10 librados: watch_flush enter 2024-09-11T05:07:52.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 10 librados: watch_flush exit 2024-09-11T05:07:52.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 20 client.4250.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 10 client.4250.objecter successfully canceled tick 2024-09-11T05:07:52.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 >> v2:172.21.15.190:6800/1562782189 conn(0x7f525c03cef0 msgr2=0x7f525c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 --2- 172.21.15.138:0/170936349 >> v2:172.21.15.190:6800/1562782189 conn(0x7f525c03cef0 0x7f525c03f3b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f527c135200 tx=0x7f52700a6000 comp rx=0 tx=0).stop 2024-09-11T05:07:52.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 10 monclient: shutdown 2024-09-11T05:07:52.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f527c070010 msgr2=0x7f527c140a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f527c070010 0x7f527c140a30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f526c00c480 tx=0x7f526c00c940 comp rx=0 tx=0).stop 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 shutdown_connections 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 --2- 172.21.15.138:0/170936349 >> v2:172.21.15.190:6800/1562782189 conn(0x7f525c03cef0 0x7f525c03f3b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f527c0a3680 0x7f527c0a3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f527c09eb10 0x7f527c0a3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 --2- 172.21.15.138:0/170936349 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f527c070010 0x7f527c140a30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 >> 172.21.15.138:0/170936349 conn(0x7f527c0ab830 msgr2=0x7f527c0aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 shutdown_connections 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 -- 172.21.15.138:0/170936349 wait complete. 2024-09-11T05:07:52.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.466+0000 7f5280fbe640 1 librados: shutdown 2024-09-11T05:07:52.485 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.0 2024-09-11T05:07:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.526+0000 7faa99f5c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T05:07:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.526+0000 7faa99f5c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T05:07:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.526+0000 7faa99f5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.526+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7faa74005380 con 0x7faa9409e6e0 2024-09-11T05:07:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.526+0000 7faa7bfff640 1 -- 172.21.15.138:0/632500069 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faa940a2330 con 0x7faa9409e6e0 2024-09-11T05:07:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.526+0000 7faa7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T05:07:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.526+0000 7faa7bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:52.532 INFO:teuthology.orchestra.run.smithi138.stdout:47244640258 2024-09-11T05:07:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 10 librados: watch_flush enter 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 10 librados: watch_flush exit 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 20 client.4260.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 10 client.4260.objecter successfully canceled tick 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 >> v2:172.21.15.190:6800/1562782189 conn(0x7faa8003cf00 msgr2=0x7faa8003f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 --2- 172.21.15.138:0/632500069 >> v2:172.21.15.190:6800/1562782189 conn(0x7faa8003cf00 0x7faa8003f3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faa840079c0 tx=0x7faa84008040 comp rx=0 tx=0).stop 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 10 monclient: shutdown 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 msgr2=0x7faa940a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa940a01b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa8801a9c0 tx=0x7faa880190b0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 shutdown_connections 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 --2- 172.21.15.138:0/632500069 >> v2:172.21.15.190:6800/1562782189 conn(0x7faa8003cf00 0x7faa8003f3c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7faa9409eb10 0x7faa940a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7faa9409e6e0 0x7faa940a01b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 --2- 172.21.15.138:0/632500069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa94070010 0x7faa94131eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 >> 172.21.15.138:0/632500069 conn(0x7faa940ab830 msgr2=0x7faa940a9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 shutdown_connections 2024-09-11T05:07:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 -- 172.21.15.138:0/632500069 wait complete. 2024-09-11T05:07:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.530+0000 7faa99f5c640 1 librados: shutdown 2024-09-11T05:07:52.551 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.4 2024-09-11T05:07:52.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T05:07:52.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T05:07:52.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbcd4069fa0 con 0x7fbcd409e6e0 2024-09-11T05:07:52.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcbb7fe640 1 -- 172.21.15.138:0/2732131836 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbcc8018dc0 con 0x7fbcd409e6e0 2024-09-11T05:07:52.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcbb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T05:07:52.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcbb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:52.601 INFO:teuthology.orchestra.run.smithi138.stdout:51539607556 2024-09-11T05:07:52.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 10 librados: watch_flush enter 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 10 librados: watch_flush exit 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 20 client.4263.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 10 client.4263.objecter successfully canceled tick 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbcc003ced0 msgr2=0x7fbcc003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 --2- 172.21.15.138:0/2732131836 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbcc003ced0 0x7fbcc003f390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4135200 tx=0x7fbcc40a6000 comp rx=0 tx=0).stop 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 10 monclient: shutdown 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbcd409e6e0 msgr2=0x7fbcd40a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbcd409e6e0 0x7fbcd40a0ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800b580 tx=0x7fbcc800ba40 comp rx=0 tx=0).stop 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 shutdown_connections 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 --2- 172.21.15.138:0/2732131836 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbcc003ced0 0x7fbcc003f390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbcd409eb10 0x7fbcd40a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbcd409e6e0 0x7fbcd40a0ae0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 --2- 172.21.15.138:0/2732131836 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbcd4070010 0x7fbcd40a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 >> 172.21.15.138:0/2732131836 conn(0x7fbcd40ab830 msgr2=0x7fbcd40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 shutdown_connections 2024-09-11T05:07:52.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.598+0000 7fbcd9583640 1 -- 172.21.15.138:0/2732131836 wait complete. 2024-09-11T05:07:52.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.602+0000 7fbcd9583640 1 librados: shutdown 2024-09-11T05:07:52.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.3 2024-09-11T05:07:52.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.622+0000 7f31f6fd3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T05:07:52.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.622+0000 7f31f6fd3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T05:07:52.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.622+0000 7f31f6fd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.622+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f31d0005650 con 0x7f31f00a5590 2024-09-11T05:07:52.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31ed7fa640 1 -- 172.21.15.138:0/2975221322 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f31e4016070 con 0x7f31f00a5590 2024-09-11T05:07:52.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T05:07:52.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:52.628 INFO:teuthology.orchestra.run.smithi138.stdout:47244640259 2024-09-11T05:07:52.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 10 librados: watch_flush enter 2024-09-11T05:07:52.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 10 librados: watch_flush exit 2024-09-11T05:07:52.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 20 client.4271.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 10 client.4271.objecter successfully canceled tick 2024-09-11T05:07:52.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31dc03cf20 msgr2=0x7f31dc03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/2975221322 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31dc03cf20 0x7f31dc03f3e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31e0007ca0 tx=0x7f31e0007bf0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 10 monclient: shutdown 2024-09-11T05:07:52.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 msgr2=0x7f31f0140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f0140f10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f31e401cbd0 tx=0x7f31e40190d0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 shutdown_connections 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/2975221322 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31dc03cf20 0x7f31dc03f3e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f013bdc0 0x7f31f0141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f00a5590 0x7f31f0140f10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 --2- 172.21.15.138:0/2975221322 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f00a4aa0 0x7f31f0142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 >> 172.21.15.138:0/2975221322 conn(0x7f31f00a9690 msgr2=0x7f31f006f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 shutdown_connections 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 -- 172.21.15.138:0/2975221322 wait complete. 2024-09-11T05:07:52.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.626+0000 7f31f6fd3640 1 librados: shutdown 2024-09-11T05:07:52.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.2 2024-09-11T05:07:52.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.642+0000 7f759cff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T05:07:52.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.642+0000 7f759cff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T05:07:52.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.642+0000 7f759cff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:52.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.642+0000 7f759cff9640 1 -- 172.21.15.138:0/2047035364 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7588005380 con 0x7f75b009e6e0 2024-09-11T05:07:52.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.642+0000 7f759effd640 1 -- 172.21.15.138:0/2047035364 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f75a40c07e0 con 0x7f75b009e6e0 2024-09-11T05:07:52.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.642+0000 7f759effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T05:07:52.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.642+0000 7f759effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:52.647 INFO:teuthology.orchestra.run.smithi138.stdout:47244640258 2024-09-11T05:07:52.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 10 librados: watch_flush enter 2024-09-11T05:07:52.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 10 librados: watch_flush exit 2024-09-11T05:07:52.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 20 client.4254.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 10 client.4254.objecter successfully canceled tick 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 -- 172.21.15.138:0/2047035364 >> v2:172.21.15.190:6800/1562782189 conn(0x7f759003cef0 msgr2=0x7f759003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 --2- 172.21.15.138:0/2047035364 >> v2:172.21.15.190:6800/1562782189 conn(0x7f759003cef0 0x7f759003f3b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009af0 tx=0x7f75a0008040 comp rx=0 tx=0).stop 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 10 monclient: shutdown 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 -- 172.21.15.138:0/2047035364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75b009e6e0 msgr2=0x7f75b013fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 --2- 172.21.15.138:0/2047035364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75b009e6e0 0x7f75b013fc60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f75a401aaa0 tx=0x7f75a4019190 comp rx=0 tx=0).stop 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 -- 172.21.15.138:0/2047035364 shutdown_connections 2024-09-11T05:07:52.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 --2- 172.21.15.138:0/2047035364 >> v2:172.21.15.190:6800/1562782189 conn(0x7f759003cef0 0x7f759003f3b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 --2- 172.21.15.138:0/2047035364 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75b01382c0 0x7f75b00a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 --2- 172.21.15.138:0/2047035364 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75b009e6e0 0x7f75b013fc60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 --2- 172.21.15.138:0/2047035364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75b0070010 0x7f75b013f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 -- 172.21.15.138:0/2047035364 >> 172.21.15.138:0/2047035364 conn(0x7f75b00ab830 msgr2=0x7f75b009d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 -- 172.21.15.138:0/2047035364 shutdown_connections 2024-09-11T05:07:52.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 -- 172.21.15.138:0/2047035364 wait complete. 2024-09-11T05:07:52.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.646+0000 7f759cff9640 1 librados: shutdown 2024-09-11T05:07:52.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.5 2024-09-11T05:07:52.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T05:07:52.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T05:07:52.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:52.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f023c005380 con 0x7f02581d0440 2024-09-11T05:07:52.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f024f7fe640 1 -- 172.21.15.138:0/1068257042 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0250014ce0 con 0x7f02581d0440 2024-09-11T05:07:52.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f024f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T05:07:52.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f024f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:52.694 INFO:teuthology.orchestra.run.smithi138.stdout:47244640258 2024-09-11T05:07:52.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 10 librados: watch_flush enter 2024-09-11T05:07:52.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 10 librados: watch_flush exit 2024-09-11T05:07:52.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 20 client.4268.objecter shutdown clearing up homeless session... 2024-09-11T05:07:52.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 10 client.4268.objecter successfully canceled tick 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 >> v2:172.21.15.190:6800/1562782189 conn(0x7f023803cef0 msgr2=0x7f023803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 --2- 172.21.15.138:0/1068257042 >> v2:172.21.15.190:6800/1562782189 conn(0x7f023803cef0 0x7f023803f3b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0248007ad0 tx=0x7f02480067e0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 10 monclient: shutdown 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02581d0440 msgr2=0x7f02581d08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02581d0440 0x7f02581d08d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f02580a22e0 tx=0x7f025000b4a0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 shutdown_connections 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 --2- 172.21.15.138:0/1068257042 >> v2:172.21.15.190:6800/1562782189 conn(0x7f023803cef0 0x7f023803f3b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f02581d0440 0x7f02581d08d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0258121cd0 0x7f02581220d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 --2- 172.21.15.138:0/1068257042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0244004810 0x7f0258121790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 >> 172.21.15.138:0/1068257042 conn(0x7f02580ab830 msgr2=0x7f02580a9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:52.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 shutdown_connections 2024-09-11T05:07:52.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 -- 172.21.15.138:0/1068257042 wait complete. 2024-09-11T05:07:52.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:52.690+0000 7f025d71e640 1 librados: shutdown 2024-09-11T05:07:52.710 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.1 2024-09-11T05:07:53.486 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T05:07:53.552 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T05:07:53.619 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T05:07:53.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.614+0000 7fa2d2d96640 10 monclient: get_monmap_and_config 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.614+0000 7fa2d2d96640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.614+0000 7fa2d2d96640 10 monclient: monmap: 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:53.621018+0000 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:53.621018+0000 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.622 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 1 Processor -- start 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 1 -- start start 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient: init 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa2cc0a3940 0x7fa2cc0a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient(hunting): picked mon.noname-a con 0x7fa2cc0a3940 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:53.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc139710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient(hunting): picked mon.noname-b con 0x7fa2cc0a4260 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2cc139c50 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2cc09ddb0 con 0x7fa2cc139c50 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fa2cc09e570 con 0x7fa2cc0a3940 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa2cc09eb30 con 0x7fa2cc0a4260 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d2d96640 10 monclient(hunting): authenticate will time out at 957.057250s 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa2cc0a3940 0x7fa2cc0a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d1333640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fa2cc139c50 0x7fa2cc13c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38424/0 (socket says 172.21.15.138:38424) 2024-09-11T05:07:53.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d1333640 1 -- 172.21.15.138:0/2525231564 learned_addr learned my addr 172.21.15.138:0/2525231564 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:53.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 10 monclient(hunting): get_auth_request con 0x7fa2cc0a3940 auth_method 0 2024-09-11T05:07:53.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient(hunting): get_auth_request con 0x7fa2cc0a4260 auth_method 0 2024-09-11T05:07:53.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 1 -- 172.21.15.138:0/2525231564 >> v2:172.21.15.138:3300/0 conn(0x7fa2cc0a3940 msgr2=0x7fa2cc0a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 1 --2- 172.21.15.138:0/2525231564 >> v2:172.21.15.138:3300/0 conn(0x7fa2cc0a3940 0x7fa2cc0a3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 1 -- 172.21.15.138:0/2525231564 >> v1:172.21.15.138:6789/0 conn(0x7fa2cc139c50 legacy=0x7fa2cc13c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 1 monclient: found mon.noname-b 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2cbfff640 1 -- 172.21.15.138:0/2525231564 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2cc09cf70 con 0x7fa2cc0a4260 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.618+0000 7fa2d0b32640 1 --2- 172.21.15.138:0/2525231564 >> v2:172.21.15.138:3300/0 conn(0x7fa2cc0a3940 0x7fa2cc0a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cbfff640 1 -- 172.21.15.138:0/2525231564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 msgr2=0x7fa2cc139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cbfff640 1 --2- 172.21.15.138:0/2525231564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc139710 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0097c0 tx=0x7fa2bc01c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe85faf422297f64 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 1 -- 172.21.15.138:0/2525231564 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc0b0070 con 0x7fa2cc0a4260 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: dump: 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.634 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 1 -- 172.21.15.138:0/2525231564 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2bc0ab600 con 0x7fa2cc0a4260 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 5 monclient: authenticate success, global_id 4274 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 4 set_mon_vals no callback set 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 1 -- 172.21.15.138:0/2525231564 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc0ab9d0 con 0x7fa2cc0a4260 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2cb7fe640 10 monclient: dump: 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.636 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 10 monclient: shutdown 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2525231564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 msgr2=0x7fa2cc139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2525231564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc139710 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0097c0 tx=0x7fa2bc01c330 comp rx=0 tx=0).stop 2024-09-11T05:07:53.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2525231564 shutdown_connections 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2525231564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc139710 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2525231564 >> v2:172.21.15.138:3300/0 conn(0x7fa2cc0a3940 0x7fa2cc0a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2525231564 >> 172.21.15.138:0/2525231564 conn(0x7fa2cc0a8530 msgr2=0x7fa2cc0a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2525231564 shutdown_connections 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2525231564 wait complete. 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.622+0000 7fa2d2d96640 10 monclient: monmap: 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:53.638 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 librados: starting msgr at 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 librados: starting objecter 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 Processor -- start 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 -- start start 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 librados: setting wanted keys 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 librados: calling monclient init 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient: init 2024-09-11T05:07:53.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2cc0a3940 0x7fa2cc1d0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient(hunting): picked mon.noname-b con 0x7fa2cc0a3940 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc1d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient(hunting): picked mon.noname-c con 0x7fa2cc0a4260 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2cc139c50 0x7fa2cc1d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient(hunting): picked mon.noname-a con 0x7fa2cc139c50 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc0a5520 con 0x7fa2cc139c50 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc0a53a0 con 0x7fa2cc0a3940 2024-09-11T05:07:53.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa2cc074840 con 0x7fa2cc0a4260 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d2d96640 10 monclient(hunting): authenticate will time out at 957.064861s 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2cc139c50 0x7fa2cc1d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2cc139c50 0x7fa2cc1d11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46408/0 (socket says 172.21.15.138:46408) 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 -- 172.21.15.138:0/2575279224 learned_addr learned my addr 172.21.15.138:0/2575279224 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient(hunting): get_auth_request con 0x7fa2cc139c50 auth_method 0 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2cbfff640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc1d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2cbfff640 10 monclient(hunting): get_auth_request con 0x7fa2cc0a4260 auth_method 0 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d0b32640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2cc0a3940 0x7fa2cc1d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d0b32640 10 monclient(hunting): get_auth_request con 0x7fa2cc0a3940 auth_method 0 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d0b32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d0b32640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d0b32640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 -- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 msgr2=0x7fa2cc1d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc1d0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 -- 172.21.15.138:0/2575279224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2cc0a3940 msgr2=0x7fa2cc1d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2cc0a3940 0x7fa2cc1d0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 monclient: found mon.noname-a 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 -- 172.21.15.138:0/2575279224 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2cc0a6520 con 0x7fa2cc139c50 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 -- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2cc139c50 msgr2=0x7fa2cc1d11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2d1333640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2cc139c50 0x7fa2cc1d11b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400caf0 tx=0x7fa2c400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2c97fa640 10 client.?.objecter ms_handle_connect 0x7fa2cc139c50 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2c97fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2c97fa640 1 -- 172.21.15.138:0/2575279224 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4012400 con 0x7fa2cc139c50 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2c97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.626+0000 7fa2c97fa640 10 monclient: dump: 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.644 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 1 -- 172.21.15.138:0/2575279224 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2c40125c0 con 0x7fa2cc139c50 2024-09-11T05:07:53.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 5 monclient: authenticate success, global_id 4278 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 1 -- 172.21.15.138:0/2575279224 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c40129b0 con 0x7fa2cc139c50 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 10 monclient: dump: 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.646 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.647 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.647 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.647 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2caffd640 4 set_mon_vals no callback set 2024-09-11T05:07:53.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 10 monclient: _renew_subs 2024-09-11T05:07:53.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2cc123a40 con 0x7fa2cc139c50 2024-09-11T05:07:53.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:53.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 10 monclient: _renew_subs 2024-09-11T05:07:53.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2cc140310 con 0x7fa2cc139c50 2024-09-11T05:07:53.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 1 librados: init done 2024-09-11T05:07:53.648 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T05:07:53.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2cc0a3d20 con 0x7fa2cc139c50 2024-09-11T05:07:53.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 1 -- 172.21.15.138:0/2575279224 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fa2c4012bd0 con 0x7fa2cc139c50 2024-09-11T05:07:53.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 1 --2- 172.21.15.138:0/2575279224 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa2b803cf00 0x7fa2b803f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 1 -- 172.21.15.138:0/2575279224 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fa2c404f8e0 con 0x7fa2cc139c50 2024-09-11T05:07:53.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 10 client.4278.objecter ms_dispatch 0x7fa2cc000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 3 client.4278.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 3 client.4278.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2c97fa640 20 client.4278.objecter dump_active .. 0 homeless 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d0b32640 1 --2- 172.21.15.138:0/2575279224 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa2b803cf00 0x7fa2b803f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.630+0000 7fa2d0b32640 10 monclient: get_auth_request con 0x7fa2b803cf00 auth_method 0 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.634+0000 7fa2c97fa640 10 client.4278.objecter ms_handle_connect 0x7fa2b803cf00 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.634+0000 7fa2d0b32640 1 --2- 172.21.15.138:0/2575279224 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa2b803cf00 0x7fa2b803f3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009580 tx=0x7fa2c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.634+0000 7fa2c97fa640 1 -- 172.21.15.138:0/2575279224 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fa2c4010d10 con 0x7fa2cc139c50 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.634+0000 7fa2c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.634+0000 7fa2c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:53.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient: get_monmap_and_config 2024-09-11T05:07:53.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient: monmap: 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:53.667532+0000 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:53.667532+0000 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:53.666 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:53.667 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:53.667 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 1 Processor -- start 2024-09-11T05:07:53.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 1 -- start start 2024-09-11T05:07:53.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient: init 2024-09-11T05:07:53.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-11T05:07:53.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff034070010 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:53.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff03409e6e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:53.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff03409eb10 0x7ff034140d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff03409eb10 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff034135b60 con 0x7ff034070010 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0340a8990 con 0x7ff03409e6e0 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff0340703f0 con 0x7ff03409eb10 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03ad9f640 10 monclient(hunting): authenticate will time out at 957.103316s 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff038b3b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7ff034070010 0x7ff034135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38434/0 (socket says 172.21.15.138:38434) 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff038b3b640 1 -- 172.21.15.138:0/755033896 learned_addr learned my addr 172.21.15.138:0/755033896 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03933c640 1 --2- 172.21.15.138:0/755033896 >> v2:172.21.15.190:3300/0 conn(0x7ff03409eb10 0x7ff034140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03933c640 10 monclient(hunting): get_auth_request con 0x7ff03409eb10 auth_method 0 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03933c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03933c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03933c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03933c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03933c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.662+0000 7ff03933c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.666+0000 7ff03933c640 10 monclient(hunting): handle_auth_done global_id 4264 payload 274 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.666+0000 7ff03933c640 1 -- 172.21.15.138:0/755033896 >> v1:172.21.15.190:6789/0 conn(0x7ff03409e6e0 legacy=0x7ff03413d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.666+0000 7ff03933c640 1 -- 172.21.15.138:0/755033896 >> v1:172.21.15.138:6789/0 conn(0x7ff034070010 legacy=0x7ff034135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.666+0000 7ff03933c640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.666+0000 7ff03933c640 1 monclient: found mon.noname-c 2024-09-11T05:07:53.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.666+0000 7ff03933c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.666+0000 7ff03933c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:07:53.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.666+0000 7ff03933c640 1 -- 172.21.15.138:0/755033896 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03409cbf0 con 0x7ff03409eb10 2024-09-11T05:07:53.672 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T05:07:53.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03933c640 1 -- 172.21.15.138:0/755033896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409eb10 msgr2=0x7ff034140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03933c640 1 --2- 172.21.15.138:0/755033896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409eb10 0x7ff034140d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009d90 tx=0x7ff02c01c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a8ff7f1997f4153 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 1 -- 172.21.15.138:0/755033896 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0b2070 con 0x7ff03409eb10 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: dump: 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.674 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.675 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.675 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.675 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.675 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 5 monclient: authenticate success, global_id 4264 2024-09-11T05:07:53.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:53.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 1 -- 172.21.15.138:0/755033896 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02c0a5920 con 0x7ff03409eb10 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 1 -- 172.21.15.138:0/755033896 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0a5cf0 con 0x7ff03409eb10 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 4 set_mon_vals no callback set 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff0337fe640 10 monclient: dump: 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.680 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 10 monclient: shutdown 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 1 -- 172.21.15.138:0/755033896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409eb10 msgr2=0x7ff034140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 1 --2- 172.21.15.138:0/755033896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409eb10 0x7ff034140d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009d90 tx=0x7ff02c01c800 comp rx=0 tx=0).stop 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 1 -- 172.21.15.138:0/755033896 shutdown_connections 2024-09-11T05:07:53.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 1 --2- 172.21.15.138:0/755033896 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409eb10 0x7ff034140d70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 1 -- 172.21.15.138:0/755033896 >> 172.21.15.138:0/755033896 conn(0x7ff0340ab830 msgr2=0x7ff0340a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 1 -- 172.21.15.138:0/755033896 shutdown_connections 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 1 -- 172.21.15.138:0/755033896 wait complete. 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.670+0000 7ff03ad9f640 10 monclient: monmap: 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.682 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 librados: starting msgr at 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 librados: starting objecter 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 Processor -- start 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 -- start start 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 librados: setting wanted keys 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 librados: calling monclient init 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient: init 2024-09-11T05:07:53.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff034070010 0x7ff0341d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff034070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409e6e0 0x7ff0341403b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient(hunting): picked mon.noname-b con 0x7ff03409e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff03409eb10 0x7ff0341408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient(hunting): picked mon.noname-a con 0x7ff03409eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff034143e60 con 0x7ff03409eb10 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff034143ce0 con 0x7ff03409e6e0 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff034143fe0 con 0x7ff034070010 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03ad9f640 10 monclient(hunting): authenticate will time out at 957.112904s 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff034070010 0x7ff0341d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff034070010 0x7ff0341d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58858/0 (socket says 172.21.15.138:58858) 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 -- 172.21.15.138:0/3745298604 learned_addr learned my addr 172.21.15.138:0/3745298604 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient(hunting): get_auth_request con 0x7ff034070010 auth_method 0 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff033fff640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409e6e0 0x7ff0341403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03933c640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff03409eb10 0x7ff0341408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 -- 172.21.15.138:0/3745298604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409e6e0 msgr2=0x7ff0341403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409e6e0 0x7ff0341403b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 -- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff03409eb10 msgr2=0x7ff0341408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff03409eb10 0x7ff0341408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 monclient: found mon.noname-c 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 -- 172.21.15.138:0/3745298604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0340aaa90 con 0x7ff034070010 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff033fff640 10 monclient: get_auth_request con 0x7ff03409e6e0 auth_method 0 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff033fff640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409e6e0 0x7ff0341403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03933c640 10 monclient: get_auth_request con 0x7ff03409eb10 auth_method 0 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff03933c640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff03409eb10 0x7ff0341408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 -- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff034070010 msgr2=0x7ff0341d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff038b3b640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff034070010 0x7ff0341d04f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff02800c470 tx=0x7ff02800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.674+0000 7ff0317fa640 1 -- 172.21.15.138:0/3745298604 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 client.?.objecter ms_handle_connect 0x7ff034070010 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 1 -- 172.21.15.138:0/3745298604 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028007070 con 0x7ff034070010 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: dump: 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.690 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 1 -- 172.21.15.138:0/3745298604 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff028007210 con 0x7ff034070010 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 5 monclient: authenticate success, global_id 4280 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 10 monclient: _renew_subs 2024-09-11T05:07:53.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff034140e30 con 0x7ff034070010 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 1 -- 172.21.15.138:0/3745298604 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028007600 con 0x7ff034070010 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 10 monclient: dump: 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.692 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff032ffd640 4 set_mon_vals no callback set 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 10 monclient: _renew_subs 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03413ae40 con 0x7ff034070010 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 1 librados: init done 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff034135450 con 0x7ff034070010 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 1 -- 172.21.15.138:0/3745298604 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7ff02801b400 con 0x7ff034070010 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.678+0000 7ff0317fa640 1 --2- 172.21.15.138:0/3745298604 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff01803cef0 0x7ff01803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.682+0000 7ff0317fa640 1 -- 172.21.15.138:0/3745298604 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7ff02804f090 con 0x7ff034070010 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.682+0000 7ff0317fa640 10 client.4280.objecter ms_dispatch 0x7ff034000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.682+0000 7ff0317fa640 3 client.4280.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.682+0000 7ff0317fa640 3 client.4280.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.682+0000 7ff0317fa640 20 client.4280.objecter dump_active .. 0 homeless 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.686+0000 7ff033fff640 1 --2- 172.21.15.138:0/3745298604 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff01803cef0 0x7ff01803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.686+0000 7ff033fff640 10 monclient: get_auth_request con 0x7ff01803cef0 auth_method 0 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.686+0000 7ff0317fa640 1 -- 172.21.15.138:0/3745298604 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7ff028012400 con 0x7ff034070010 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.686+0000 7ff0317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.686+0000 7ff0317fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:53.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.686+0000 7ff0317fa640 10 client.4280.objecter ms_handle_connect 0x7ff01803cef0 2024-09-11T05:07:53.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.686+0000 7ff033fff640 1 --2- 172.21.15.138:0/3745298604 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff01803cef0 0x7ff01803f3b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff0341398c0 tx=0x7ff020009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.712 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.726+0000 7f17ca26e640 10 monclient: get_monmap_and_config 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.726+0000 7f17ca26e640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.726+0000 7f17ca26e640 10 monclient: monmap: 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:53.732810+0000 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:53.732810+0000 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.731 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:53.732 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:53.732 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:53.732 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:53.732 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:53.732 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 1 Processor -- start 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 1 -- start start 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient: init 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f17c40a62a0 0x7f17c40a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient(hunting): picked mon.noname-a con 0x7f17c40a62a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f17c406ef10 0x7f17c40a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient(hunting): picked mon.noname-c con 0x7f17c406ef10 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a0850 0x7f17c40a5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient(hunting): picked mon.noname-b con 0x7f17c40a0850 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f17c40a6680 con 0x7f17c40a62a0 2024-09-11T05:07:53.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f17c40709f0 con 0x7f17c406ef10 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f17c406ecd0 con 0x7f17c40a0850 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17ca26e640 10 monclient(hunting): authenticate will time out at 957.169196s 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a0850 0x7f17c40a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a0850 0x7f17c40a5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58870/0 (socket says 172.21.15.138:58870) 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 -- 172.21.15.138:0/2958918061 learned_addr learned my addr 172.21.15.138:0/2958918061 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient(hunting): get_auth_request con 0x7f17c40a0850 auth_method 0 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17bbfff640 1 --2- 172.21.15.138:0/2958918061 >> v2:172.21.15.190:3300/0 conn(0x7f17c406ef10 0x7f17c40a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17bbfff640 10 monclient(hunting): get_auth_request con 0x7f17c406ef10 auth_method 0 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 -- 172.21.15.138:0/2958918061 >> v2:172.21.15.190:3300/0 conn(0x7f17c406ef10 msgr2=0x7f17c40a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 --2- 172.21.15.138:0/2958918061 >> v2:172.21.15.190:3300/0 conn(0x7f17c406ef10 0x7f17c40a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 -- 172.21.15.138:0/2958918061 >> v2:172.21.15.138:3300/0 conn(0x7f17c40a62a0 msgr2=0x7f17c40a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c3fff640 1 --2- 172.21.15.138:0/2958918061 >> v2:172.21.15.138:3300/0 conn(0x7f17c40a62a0 0x7f17c40a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 --2- 172.21.15.138:0/2958918061 >> v2:172.21.15.138:3300/0 conn(0x7f17c40a62a0 0x7f17c40a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 monclient: found mon.noname-b 2024-09-11T05:07:53.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c880b640 1 -- 172.21.15.138:0/2958918061 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c4070430 con 0x7f17c40a0850 2024-09-11T05:07:53.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c3fff640 10 monclient: get_auth_request con 0x7f17c40a62a0 auth_method 0 2024-09-11T05:07:53.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.730+0000 7f17c3fff640 1 --2- 172.21.15.138:0/2958918061 >> v2:172.21.15.138:3300/0 conn(0x7f17c40a62a0 0x7f17c40a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:53.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c880b640 1 -- 172.21.15.138:0/2958918061 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a0850 msgr2=0x7f17c40a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c880b640 1 --2- 172.21.15.138:0/2958918061 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a0850 0x7f17c40a5520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17c40a01e0 tx=0x7f17bc01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd4c2e0eb399ccb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c37fe640 1 -- 172.21.15.138:0/2958918061 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0a5640 con 0x7f17c40a0850 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c37fe640 10 monclient: dump: 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.740 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c37fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c37fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.734+0000 7f17c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17c37fe640 1 -- 172.21.15.138:0/2958918061 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc0a57e0 con 0x7f17c40a0850 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17c37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17c37fe640 1 -- 172.21.15.138:0/2958918061 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0a5bb0 con 0x7f17c40a0850 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17c37fe640 10 monclient: dump: 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.741 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 5 monclient: authenticate success, global_id 4283 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:53.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 4 set_mon_vals no callback set 2024-09-11T05:07:53.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 10 monclient: shutdown 2024-09-11T05:07:53.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 -- 172.21.15.138:0/2958918061 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a0850 msgr2=0x7f17c40a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 --2- 172.21.15.138:0/2958918061 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a0850 0x7f17c40a5520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f17c40a01e0 tx=0x7f17bc01c8b0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 -- 172.21.15.138:0/2958918061 shutdown_connections 2024-09-11T05:07:53.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 --2- 172.21.15.138:0/2958918061 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a0850 0x7f17c40a5520 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 --2- 172.21.15.138:0/2958918061 >> v2:172.21.15.190:3300/0 conn(0x7f17c406ef10 0x7f17c40a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 --2- 172.21.15.138:0/2958918061 >> v2:172.21.15.138:3300/0 conn(0x7f17c40a62a0 0x7f17c40a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 -- 172.21.15.138:0/2958918061 >> 172.21.15.138:0/2958918061 conn(0x7f17c40a9690 msgr2=0x7f17c40a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 -- 172.21.15.138:0/2958918061 shutdown_connections 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 -- 172.21.15.138:0/2958918061 wait complete. 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 10 monclient: monmap: 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.747 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 librados: starting msgr at 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 librados: starting objecter 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 Processor -- start 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 -- start start 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 librados: setting wanted keys 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 1 librados: calling monclient init 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.738+0000 7f17ca26e640 10 monclient: init 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17c406ef10 0x7f17c4123fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient(hunting): picked mon.noname-a con 0x7f17c406ef10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c40a0850 0x7f17c4125f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient(hunting): picked mon.noname-b con 0x7f17c40a0850 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a62a0 0x7f17c413bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient(hunting): picked mon.noname-c con 0x7f17c40a62a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4141880 con 0x7f17c406ef10 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4141700 con 0x7f17c40a0850 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a62a0 0x7f17c413bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a62a0 0x7f17c413bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58882/0 (socket says 172.21.15.138:58882) 2024-09-11T05:07:53.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 -- 172.21.15.138:0/1162584785 learned_addr learned my addr 172.21.15.138:0/1162584785 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f17c4141a00 con 0x7f17c40a62a0 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c3fff640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17c406ef10 0x7f17c4123fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c40a0850 0x7f17c4125f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17ca26e640 10 monclient(hunting): authenticate will time out at 957.181335s 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient(hunting): get_auth_request con 0x7f17c40a62a0 auth_method 0 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c3fff640 10 monclient(hunting): get_auth_request con 0x7f17c406ef10 auth_method 0 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 10 monclient(hunting): get_auth_request con 0x7f17c40a0850 auth_method 0 2024-09-11T05:07:53.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 -- 172.21.15.138:0/1162584785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c40a0850 msgr2=0x7f17c4125f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c40a0850 0x7f17c4125f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 -- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17c406ef10 msgr2=0x7f17c4123fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17c406ef10 0x7f17c4123fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 monclient: found mon.noname-c 2024-09-11T05:07:53.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17c880b640 1 -- 172.21.15.138:0/1162584785 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c40a85f0 con 0x7f17c40a62a0 2024-09-11T05:07:53.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.742+0000 7f17bbfff640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c40a0850 0x7f17c4125f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:53.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c880b640 1 -- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a62a0 msgr2=0x7f17c413bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c880b640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a62a0 0x7f17c413bda0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f17bc01d150 tx=0x7f17bc019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 client.?.objecter ms_handle_connect 0x7f17c40a62a0 2024-09-11T05:07:53.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:53.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 1 -- 172.21.15.138:0/1162584785 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0a5640 con 0x7f17c40a62a0 2024-09-11T05:07:53.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: dump: 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 5 monclient: authenticate success, global_id 4286 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 10 monclient: _renew_subs 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c40a0280 con 0x7f17c40a62a0 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 10 monclient: _renew_subs 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c413c2e0 con 0x7f17c40a62a0 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 1 librados: init done 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 1 -- 172.21.15.138:0/1162584785 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc0bc070 con 0x7f17c40a62a0 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 1 -- 172.21.15.138:0/1162584785 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0b6d00 con 0x7f17c40a62a0 2024-09-11T05:07:53.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 monclient: dump: 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c2ffd640 4 set_mon_vals no callback set 2024-09-11T05:07:53.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 1 -- 172.21.15.138:0/1162584785 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f17bc0c1460 con 0x7f17c40a62a0 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 1 --2- 172.21.15.138:0/1162584785 >> v2:172.21.15.190:6800/1562782189 conn(0x7f17a403cea0 0x7f17a403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 1 -- 172.21.15.138:0/1162584785 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f17bc0ea120 con 0x7f17c40a62a0 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 10 client.4286.objecter ms_dispatch 0x7f17c4000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 3 client.4286.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 3 client.4286.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c17fa640 20 client.4286.objecter dump_active .. 0 homeless 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c406f6d0 con 0x7f17c40a62a0 2024-09-11T05:07:53.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c3fff640 1 --2- 172.21.15.138:0/1162584785 >> v2:172.21.15.190:6800/1562782189 conn(0x7f17a403cea0 0x7f17a403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.746+0000 7f17c3fff640 10 monclient: get_auth_request con 0x7f17a403cea0 auth_method 0 2024-09-11T05:07:53.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.750+0000 7f17c3fff640 1 --2- 172.21.15.138:0/1162584785 >> v2:172.21.15.190:6800/1562782189 conn(0x7f17a403cea0 0x7f17a403f360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f17b400aae0 tx=0x7f17b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.754+0000 7f17c17fa640 10 client.4286.objecter ms_handle_connect 0x7f17a403cea0 2024-09-11T05:07:53.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.758+0000 7f17c17fa640 1 -- 172.21.15.138:0/1162584785 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f17c40a85f0 con 0x7f17c40a62a0 2024-09-11T05:07:53.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.758+0000 7f17c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.758+0000 7f17c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:53.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient: get_monmap_and_config 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient: monmap: 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:53.765742+0000 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:53.765742+0000 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:53.764 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:53.765 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:53.765 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 1 Processor -- start 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 1 -- start start 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient: init 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1dd8070010 0x7f1dd8135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient(hunting): picked mon.noname-a con 0x7f1dd8070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1dd809e6e0 0x7f1dd81382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient(hunting): picked mon.noname-c con 0x7f1dd809e6e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1dd809eb10 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:53.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1dd8070010 0x7f1dd8135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dd80a8990 con 0x7f1dd809eb10 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1dd8070010 0x7f1dd8135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46436/0 (socket says 172.21.15.138:46436) 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 1 -- 172.21.15.138:0/4020587714 learned_addr learned my addr 172.21.15.138:0/4020587714 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 1 -- 172.21.15.138:0/4020587714 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f1dd80703f0 con 0x7f1dd8070010 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 1 -- 172.21.15.138:0/4020587714 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1dd809cbf0 con 0x7f1dd809e6e0 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1de06e8640 10 monclient(hunting): authenticate will time out at 957.202097s 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient(hunting): get_auth_request con 0x7f1dd8070010 auth_method 0 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dddc83640 1 --2- 172.21.15.138:0/4020587714 >> v2:172.21.15.190:3300/0 conn(0x7f1dd809e6e0 0x7f1dd81382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dddc83640 10 monclient(hunting): get_auth_request con 0x7f1dd809e6e0 auth_method 0 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dddc83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dddc83640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dddc83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dddc83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dddc83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dddc83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 1 -- 172.21.15.138:0/4020587714 >> v2:172.21.15.190:3300/0 conn(0x7f1dd809e6e0 msgr2=0x7f1dd81382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 1 --2- 172.21.15.138:0/4020587714 >> v2:172.21.15.190:3300/0 conn(0x7f1dd809e6e0 0x7f1dd81382c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 1 -- 172.21.15.138:0/4020587714 >> v1:172.21.15.190:6789/0 conn(0x7f1dd809eb10 legacy=0x7f1dd8138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 1 monclient: found mon.noname-a 2024-09-11T05:07:53.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.762+0000 7f1dde484640 1 -- 172.21.15.138:0/4020587714 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd809ced0 con 0x7f1dd8070010 2024-09-11T05:07:53.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1dde484640 1 -- 172.21.15.138:0/4020587714 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd8070010 msgr2=0x7f1dd8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1dde484640 1 --2- 172.21.15.138:0/4020587714 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd8070010 0x7f1dd8135450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1dd806efb0 tx=0x7f1dd401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ec496529de9e9b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 1 -- 172.21.15.138:0/4020587714 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40b2070 con 0x7f1dd8070010 2024-09-11T05:07:53.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:53.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: dump: 2024-09-11T05:07:53.772 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.772 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.772 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 1 -- 172.21.15.138:0/4020587714 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dd40ad640 con 0x7f1dd8070010 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 5 monclient: authenticate success, global_id 4284 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:53.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 1 -- 172.21.15.138:0/4020587714 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40ada10 con 0x7f1dd8070010 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddd482640 10 monclient: dump: 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:53.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 4 set_mon_vals no callback set 2024-09-11T05:07:53.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient: shutdown 2024-09-11T05:07:53.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- 172.21.15.138:0/4020587714 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd8070010 msgr2=0x7f1dd8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 --2- 172.21.15.138:0/4020587714 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd8070010 0x7f1dd8135450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1dd806efb0 tx=0x7f1dd401c3b0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- 172.21.15.138:0/4020587714 shutdown_connections 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 --2- 172.21.15.138:0/4020587714 >> v2:172.21.15.190:3300/0 conn(0x7f1dd809e6e0 0x7f1dd81382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 --2- 172.21.15.138:0/4020587714 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd8070010 0x7f1dd8135450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- 172.21.15.138:0/4020587714 >> 172.21.15.138:0/4020587714 conn(0x7f1dd80ab830 msgr2=0x7f1dd80a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- 172.21.15.138:0/4020587714 shutdown_connections 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- 172.21.15.138:0/4020587714 wait complete. 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient: monmap: 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.778 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 librados: starting msgr at 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 librados: starting objecter 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 Processor -- start 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- start start 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 librados: setting wanted keys 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 librados: calling monclient init 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient: init 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:53.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd8070010 0x7f1dd81d0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient(hunting): picked mon.noname-b con 0x7f1dd8070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1dd809e6e0 0x7f1dd81d0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient(hunting): picked mon.noname-c con 0x7f1dd809e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd809eb10 0x7f1dd81d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient(hunting): picked mon.noname-a con 0x7f1dd809eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8142e80 con 0x7f1dd809eb10 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd8139120 con 0x7f1dd8070010 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1dd8143000 con 0x7f1dd809e6e0 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1de06e8640 10 monclient(hunting): authenticate will time out at 957.211787s 2024-09-11T05:07:53.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1dddc83640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1dd809e6e0 0x7f1dd81d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddec85640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd809eb10 0x7f1dd81d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddec85640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd809eb10 0x7f1dd81d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46446/0 (socket says 172.21.15.138:46446) 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1dddc83640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1dd809e6e0 0x7f1dd81d0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58884/0 (socket says 172.21.15.138:58884) 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddec85640 1 -- 172.21.15.138:0/3733837468 learned_addr learned my addr 172.21.15.138:0/3733837468 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddec85640 10 monclient(hunting): get_auth_request con 0x7f1dd809eb10 auth_method 0 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddec85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddec85640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.770+0000 7f1ddec85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dddc83640 10 monclient(hunting): get_auth_request con 0x7f1dd809e6e0 auth_method 0 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dddc83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dddc83640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dddc83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dde484640 1 --2- 172.21.15.138:0/3733837468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd8070010 0x7f1dd81d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dde484640 10 monclient(hunting): get_auth_request con 0x7f1dd8070010 auth_method 0 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dde484640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dde484640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dde484640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dddc83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dddc83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dddc83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dde484640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dde484640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dde484640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2024-09-11T05:07:53.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 1 -- 172.21.15.138:0/3733837468 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1dd809e6e0 msgr2=0x7f1dd81d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 1 --2- 172.21.15.138:0/3733837468 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1dd809e6e0 0x7f1dd81d0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 1 -- 172.21.15.138:0/3733837468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd8070010 msgr2=0x7f1dd81d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 1 --2- 172.21.15.138:0/3733837468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd8070010 0x7f1dd81d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 1 monclient: found mon.noname-a 2024-09-11T05:07:53.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 1 -- 172.21.15.138:0/3733837468 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd80a1d10 con 0x7f1dd809eb10 2024-09-11T05:07:53.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 1 -- 172.21.15.138:0/3733837468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd809eb10 msgr2=0x7f1dd81d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 client.?.objecter ms_handle_connect 0x7f1dd809eb10 2024-09-11T05:07:53.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddec85640 1 --2- 172.21.15.138:0/3733837468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd809eb10 0x7f1dd81d0ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c780 tx=0x7f1dd000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:53.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 1 -- 172.21.15.138:0/3733837468 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd000abe0 con 0x7f1dd809eb10 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: dump: 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 1 -- 172.21.15.138:0/3733837468 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dd000ad80 con 0x7f1dd809eb10 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1ddcc81640 4 set_mon_vals no callback set 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 1 -- 172.21.15.138:0/3733837468 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0004bb0 con 0x7f1dd809eb10 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1dceffd640 10 monclient: dump: 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.787 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1de06e8640 5 monclient: authenticate success, global_id 4287 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1de06e8640 10 monclient: _renew_subs 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1de06e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.774+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd80a2c00 con 0x7f1dd809eb10 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 10 monclient: _renew_subs 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd80a4b80 con 0x7f1dd809eb10 2024-09-11T05:07:53.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 1 librados: init done 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd8135450 con 0x7f1dd809eb10 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1dceffd640 1 -- 172.21.15.138:0/3733837468 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1dd0005030 con 0x7f1dd809eb10 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1dceffd640 1 --2- 172.21.15.138:0/3733837468 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1dbc03cef0 0x7f1dbc03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1dde484640 1 --2- 172.21.15.138:0/3733837468 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1dbc03cef0 0x7f1dbc03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.778+0000 7f1dde484640 10 monclient: get_auth_request con 0x7f1dbc03cef0 auth_method 0 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.782+0000 7f1dceffd640 10 client.4287.objecter ms_handle_connect 0x7f1dbc03cef0 2024-09-11T05:07:53.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.782+0000 7f1dde484640 1 --2- 172.21.15.138:0/3733837468 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1dbc03cef0 0x7f1dbc03f3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1dd40ac000 tx=0x7f1dd40a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.786+0000 7f1dceffd640 1 -- 172.21.15.138:0/3733837468 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f1dd0054060 con 0x7f1dd809eb10 2024-09-11T05:07:53.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.786+0000 7f1dceffd640 10 client.4287.objecter ms_dispatch 0x7f1dd8000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:53.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.786+0000 7f1dceffd640 3 client.4287.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:53.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.786+0000 7f1dceffd640 3 client.4287.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:53.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.786+0000 7f1dceffd640 20 client.4287.objecter dump_active .. 0 homeless 2024-09-11T05:07:53.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.786+0000 7f1dceffd640 1 -- 172.21.15.138:0/3733837468 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f1dd0005350 con 0x7f1dd809eb10 2024-09-11T05:07:53.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.786+0000 7f1dceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.786+0000 7f1dceffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:53.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.838+0000 7fd4cf336640 10 monclient: get_monmap_and_config 2024-09-11T05:07:53.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.838+0000 7fd4cf336640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.838+0000 7fd4cf336640 10 monclient: monmap: 2024-09-11T05:07:53.843 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.843 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.843 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:53.844545+0000 2024-09-11T05:07:53.843 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:53.844545+0000 2024-09-11T05:07:53.843 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.844 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.844 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:53.844 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:53.844 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:53.844 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:53.844 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:53.844 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.838+0000 7fd4cf336640 1 Processor -- start 2024-09-11T05:07:53.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 1 -- start start 2024-09-11T05:07:53.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient: init 2024-09-11T05:07:53.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd4c8070010 0x7fd4c8135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient(hunting): picked mon.noname-a con 0x7fd4c8070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4c809e6e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd4c809eb10 0x7fd4c81389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient(hunting): picked mon.noname-c con 0x7fd4c809eb10 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4c80a8990 con 0x7fd4c809e6e0 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fd4c80703f0 con 0x7fd4c8070010 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd4c8070010 0x7fd4c8135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd4c8070010 0x7fd4c8135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46452/0 (socket says 172.21.15.138:46452) 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 1 -- 172.21.15.138:0/1506631034 learned_addr learned my addr 172.21.15.138:0/1506631034 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 1 -- 172.21.15.138:0/1506631034 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd4c809cbf0 con 0x7fd4c809eb10 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cf336640 10 monclient(hunting): authenticate will time out at 957.281630s 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient(hunting): get_auth_request con 0x7fd4c8070010 auth_method 0 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd8d3640 1 --2- 172.21.15.138:0/1506631034 >> v2:172.21.15.190:3300/0 conn(0x7fd4c809eb10 0x7fd4c81389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd8d3640 10 monclient(hunting): get_auth_request con 0x7fd4c809eb10 auth_method 0 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd8d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd8d3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd8d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd8d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd8d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd8d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient(hunting): handle_auth_done global_id 4290 payload 274 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 1 -- 172.21.15.138:0/1506631034 >> v2:172.21.15.190:3300/0 conn(0x7fd4c809eb10 msgr2=0x7fd4c81389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 1 --2- 172.21.15.138:0/1506631034 >> v2:172.21.15.190:3300/0 conn(0x7fd4c809eb10 0x7fd4c81389d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 1 -- 172.21.15.138:0/1506631034 >> v1:172.21.15.190:6789/0 conn(0x7fd4c809e6e0 legacy=0x7fd4c81382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 1 monclient: found mon.noname-a 2024-09-11T05:07:53.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.842+0000 7fd4cd0d2640 1 -- 172.21.15.138:0/1506631034 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c809ced0 con 0x7fd4c8070010 2024-09-11T05:07:53.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4cd0d2640 1 -- 172.21.15.138:0/1506631034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c8070010 msgr2=0x7fd4c8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4cd0d2640 1 --2- 172.21.15.138:0/1506631034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c8070010 0x7fd4c8135450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8135200 tx=0x7fd4c401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c91d753e35ae82 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 1 -- 172.21.15.138:0/1506631034 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c40b2070 con 0x7fd4c8070010 2024-09-11T05:07:53.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: dump: 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.852 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 1 -- 172.21.15.138:0/1506631034 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c40ad640 con 0x7fd4c8070010 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 1 -- 172.21.15.138:0/1506631034 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c40ada10 con 0x7fd4c8070010 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.846+0000 7fd4bffff640 10 monclient: dump: 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 5 monclient: authenticate success, global_id 4290 2024-09-11T05:07:53.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:53.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 4 set_mon_vals no callback set 2024-09-11T05:07:53.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient: shutdown 2024-09-11T05:07:53.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- 172.21.15.138:0/1506631034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c8070010 msgr2=0x7fd4c8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 --2- 172.21.15.138:0/1506631034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c8070010 0x7fd4c8135450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8135200 tx=0x7fd4c401c3b0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- 172.21.15.138:0/1506631034 shutdown_connections 2024-09-11T05:07:53.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 --2- 172.21.15.138:0/1506631034 >> v2:172.21.15.190:3300/0 conn(0x7fd4c809eb10 0x7fd4c81389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 --2- 172.21.15.138:0/1506631034 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c8070010 0x7fd4c8135450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- 172.21.15.138:0/1506631034 >> 172.21.15.138:0/1506631034 conn(0x7fd4c80ab830 msgr2=0x7fd4c80a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- 172.21.15.138:0/1506631034 shutdown_connections 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- 172.21.15.138:0/1506631034 wait complete. 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient: monmap: 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:53.858 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 librados: starting msgr at 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 librados: starting objecter 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 Processor -- start 2024-09-11T05:07:53.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- start start 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 librados: setting wanted keys 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 librados: calling monclient init 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient: init 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4c8070010 0x7fd4c80a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient(hunting): picked mon.noname-b con 0x7fd4c8070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c809e6e0 0x7fd4c80a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient(hunting): picked mon.noname-a con 0x7fd4c809e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4c809eb10 0x7fd4c80a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient(hunting): picked mon.noname-c con 0x7fd4c809eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c8142e80 con 0x7fd4c809e6e0 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c8139120 con 0x7fd4c8070010 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd4c8143000 con 0x7fd4c809eb10 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cf336640 10 monclient(hunting): authenticate will time out at 957.291763s 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cc8d1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c809e6e0 0x7fd4c80a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cd8d3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4c809eb10 0x7fd4c80a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cc8d1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c809e6e0 0x7fd4c80a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46466/0 (socket says 172.21.15.138:46466) 2024-09-11T05:07:53.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cc8d1640 1 -- 172.21.15.138:0/2879720024 learned_addr learned my addr 172.21.15.138:0/2879720024 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cc8d1640 10 monclient(hunting): get_auth_request con 0x7fd4c809e6e0 auth_method 0 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cc8d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cc8d1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cc8d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cd8d3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4c809eb10 0x7fd4c80a0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58886/0 (socket says 172.21.15.138:58886) 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cd8d3640 10 monclient(hunting): get_auth_request con 0x7fd4c809eb10 auth_method 0 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cd8d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cd8d3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.850+0000 7fd4cd8d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cd8d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cd8d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cd8d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cd0d2640 1 --2- 172.21.15.138:0/2879720024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4c8070010 0x7fd4c80a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cd0d2640 10 monclient(hunting): get_auth_request con 0x7fd4c8070010 auth_method 0 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cd0d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cd0d2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cd0d2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-09-11T05:07:53.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 1 -- 172.21.15.138:0/2879720024 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4c809eb10 msgr2=0x7fd4c80a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 1 --2- 172.21.15.138:0/2879720024 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4c809eb10 0x7fd4c80a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 1 -- 172.21.15.138:0/2879720024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4c8070010 msgr2=0x7fd4c80a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 1 --2- 172.21.15.138:0/2879720024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4c8070010 0x7fd4c80a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 1 monclient: found mon.noname-a 2024-09-11T05:07:53.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 1 -- 172.21.15.138:0/2879720024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c80a3a30 con 0x7fd4c809e6e0 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 1 -- 172.21.15.138:0/2879720024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c809e6e0 msgr2=0x7fd4c80a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cc8d1640 1 --2- 172.21.15.138:0/2879720024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c809e6e0 0x7fd4c80a0a60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c450 tx=0x7fd4b800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 client.?.objecter ms_handle_connect 0x7fd4c809e6e0 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 1 -- 172.21.15.138:0/2879720024 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80119a0 con 0x7fd4c809e6e0 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: dump: 2024-09-11T05:07:53.866 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 1 -- 172.21.15.138:0/2879720024 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4b8017070 con 0x7fd4c809e6e0 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 5 monclient: authenticate success, global_id 4293 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 10 monclient: _renew_subs 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c80a3d10 con 0x7fd4c809e6e0 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 1 -- 172.21.15.138:0/2879720024 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8011da0 con 0x7fd4c809e6e0 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bdffb640 10 monclient: dump: 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.868 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4bf7fe640 4 set_mon_vals no callback set 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 10 monclient: _renew_subs 2024-09-11T05:07:53.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c80a2690 con 0x7fd4c809e6e0 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.854+0000 7fd4cf336640 1 librados: init done 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4bdffb640 1 -- 172.21.15.138:0/2879720024 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fd4b801c460 con 0x7fd4c809e6e0 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4bdffb640 1 --2- 172.21.15.138:0/2879720024 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4ac03cef0 0x7fd4ac03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4bdffb640 1 -- 172.21.15.138:0/2879720024 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7fd4b8052120 con 0x7fd4c809e6e0 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4bdffb640 10 client.4293.objecter ms_dispatch 0x7fd4c8000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4bdffb640 3 client.4293.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4bdffb640 3 client.4293.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4bdffb640 20 client.4293.objecter dump_active .. 0 homeless 2024-09-11T05:07:53.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4cf336640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4cf336640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4cf336640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c8135450 con 0x7fd4c809e6e0 2024-09-11T05:07:53.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4cd0d2640 1 --2- 172.21.15.138:0/2879720024 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4ac03cef0 0x7fd4ac03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.858+0000 7fd4cd0d2640 10 monclient: get_auth_request con 0x7fd4ac03cef0 auth_method 0 2024-09-11T05:07:53.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.862+0000 7fd4bdffb640 10 client.4293.objecter ms_handle_connect 0x7fd4ac03cef0 2024-09-11T05:07:53.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.862+0000 7fd4cd0d2640 1 --2- 172.21.15.138:0/2879720024 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4ac03cef0 0x7fd4ac03f3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40ac000 tx=0x7fd4c40a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 10 monclient: get_monmap_and_config 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 10 monclient: monmap: 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:53.871548+0000 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:53.871548+0000 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.872 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:53.873 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:53.873 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:53.873 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:53.873 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:53.873 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7fd4bdffb640 1 -- 172.21.15.138:0/2879720024 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fd4b8019cb0 con 0x7fd4c809e6e0 2024-09-11T05:07:53.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7fd4bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7fd4bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:53.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 1 Processor -- start 2024-09-11T05:07:53.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 1 -- start start 2024-09-11T05:07:53.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 10 monclient: init 2024-09-11T05:07:53.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-11T05:07:53.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b440bfe70 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:53.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2b440254c0 0x7f2b440c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 10 monclient(hunting): picked mon.noname-c con 0x7f2b440254c0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.866+0000 7f2b52ea6640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2b440c51a0 0x7f2b440c7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b52ea6640 10 monclient(hunting): picked mon.noname-a con 0x7f2b440c51a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2b440c51a0 0x7f2b440c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2b440c51a0 0x7f2b440c7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46468/0 (socket says 172.21.15.138:46468) 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 1 -- 172.21.15.138:0/1302456135 learned_addr learned my addr 172.21.15.138:0/1302456135 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b52ea6640 1 -- 172.21.15.138:0/1302456135 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b44022920 con 0x7f2b440bfe70 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b52ea6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b52ea6640 1 -- 172.21.15.138:0/1302456135 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f2b44024490 con 0x7f2b440c51a0 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b52ea6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b52ea6640 1 -- 172.21.15.138:0/1302456135 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2b44021520 con 0x7f2b440254c0 2024-09-11T05:07:53.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b52ea6640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b52ea6640 10 monclient(hunting): authenticate will time out at 957.308337s 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient(hunting): get_auth_request con 0x7f2b440c51a0 auth_method 0 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b516a3640 1 --2- 172.21.15.138:0/1302456135 >> v2:172.21.15.190:3300/0 conn(0x7f2b440254c0 0x7f2b440c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b516a3640 10 monclient(hunting): get_auth_request con 0x7f2b440254c0 auth_method 0 2024-09-11T05:07:53.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b516a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b516a3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b516a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 1 -- 172.21.15.138:0/1302456135 >> v2:172.21.15.190:3300/0 conn(0x7f2b440254c0 msgr2=0x7f2b440c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 1 --2- 172.21.15.138:0/1302456135 >> v2:172.21.15.190:3300/0 conn(0x7f2b440254c0 0x7f2b440c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 1 -- 172.21.15.138:0/1302456135 >> v1:172.21.15.138:6789/0 conn(0x7f2b440bfe70 legacy=0x7f2b440c0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 1 monclient: found mon.noname-a 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.870+0000 7f2b526a5640 1 -- 172.21.15.138:0/1302456135 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b44022da0 con 0x7f2b440c51a0 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b526a5640 1 -- 172.21.15.138:0/1302456135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440c51a0 msgr2=0x7f2b440c7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b526a5640 1 --2- 172.21.15.138:0/1302456135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440c51a0 0x7f2b440c7590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c066720 tx=0x7f2b4c099bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82de3bbd8d02485 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 1 -- 172.21.15.138:0/1302456135 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c12b020 con 0x7f2b440c51a0 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: dump: 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.899 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 1 -- 172.21.15.138:0/1302456135 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b4c127640 con 0x7f2b440c51a0 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 5 monclient: authenticate success, global_id 4296 2024-09-11T05:07:53.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 4 set_mon_vals no callback set 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 1 -- 172.21.15.138:0/1302456135 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c123040 con 0x7f2b440c51a0 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b50ea2640 10 monclient: dump: 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.901 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 10 monclient: shutdown 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 1 -- 172.21.15.138:0/1302456135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440c51a0 msgr2=0x7f2b440c7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/1302456135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440c51a0 0x7f2b440c7590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c066720 tx=0x7f2b4c099bb0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 1 -- 172.21.15.138:0/1302456135 shutdown_connections 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/1302456135 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440c51a0 0x7f2b440c7590 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/1302456135 >> v2:172.21.15.190:3300/0 conn(0x7f2b440254c0 0x7f2b440c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 1 -- 172.21.15.138:0/1302456135 >> 172.21.15.138:0/1302456135 conn(0x7f2b44006320 msgr2=0x7f2b44006730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 1 -- 172.21.15.138:0/1302456135 shutdown_connections 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.874+0000 7f2b52ea6640 1 -- 172.21.15.138:0/1302456135 wait complete. 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient: build_initial_monmap 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient: monmap: 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:53.903 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 librados: starting msgr at 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 librados: starting objecter 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 Processor -- start 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 -- start start 2024-09-11T05:07:53.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 librados: setting wanted keys 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 librados: calling monclient init 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient: init 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2b440254c0 0x7f2b440bdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient(hunting): picked mon.noname-c con 0x7f2b440254c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440bfe70 0x7f2b440be130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2b440254c0 0x7f2b440bdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2b440254c0 0x7f2b440bdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58894/0 (socket says 172.21.15.138:58894) 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient(hunting): picked mon.noname-a con 0x7f2b440bfe70 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b440c51a0 0x7f2b4401d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440bfe70 0x7f2b440be130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440bfe70 0x7f2b440be130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46484/0 (socket says 172.21.15.138:46484) 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient(hunting): picked mon.noname-b con 0x7f2b440c51a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2b440ca470 con 0x7f2b440bfe70 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b440ca2f0 con 0x7f2b440c51a0 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2b440ca5f0 con 0x7f2b440254c0 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b52ea6640 10 monclient(hunting): authenticate will time out at 957.318560s 2024-09-11T05:07:53.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 1 -- 172.21.15.138:0/2143061470 learned_addr learned my addr 172.21.15.138:0/2143061470 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient(hunting): get_auth_request con 0x7f2b440bfe70 auth_method 0 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 10 monclient(hunting): get_auth_request con 0x7f2b440254c0 auth_method 0 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:53.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b526a5640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b440c51a0 0x7f2b4401d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b526a5640 10 monclient(hunting): get_auth_request con 0x7f2b440c51a0 auth_method 0 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b526a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b526a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b526a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 1 -- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2b440254c0 msgr2=0x7f2b440bdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2b440254c0 0x7f2b440bdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 1 -- 172.21.15.138:0/2143061470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b440c51a0 msgr2=0x7f2b4401d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b440c51a0 0x7f2b4401d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient: _finish_hunting 0 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 1 monclient: found mon.noname-a 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b516a3640 1 -- 172.21.15.138:0/2143061470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b44020e00 con 0x7f2b440bfe70 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b526a5640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b440c51a0 0x7f2b4401d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.878+0000 7f2b51ea4640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2b440254c0 0x7f2b440bdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b516a3640 1 -- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440bfe70 msgr2=0x7f2b440be130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b516a3640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440bfe70 0x7f2b440be130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00d620 tx=0x7f2b3c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 client.?.objecter ms_handle_connect 0x7f2b440bfe70 2024-09-11T05:07:53.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 1 -- 172.21.15.138:0/2143061470 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c018020 con 0x7f2b440bfe70 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: dump: 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.910 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: _finish_auth 0 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: _check_auth_tickets 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 1 -- 172.21.15.138:0/2143061470 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b3c014830 con 0x7f2b440bfe70 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b52ea6640 5 monclient: authenticate success, global_id 4299 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b52ea6640 10 monclient: _renew_subs 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b52ea6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b440210e0 con 0x7f2b440bfe70 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 1 -- 172.21.15.138:0/2143061470 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c014c00 con 0x7f2b440bfe70 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 10 monclient: dump: 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:53.912 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b43fff640 4 set_mon_vals no callback set 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 1 -- 172.21.15.138:0/2143061470 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f2b3c013050 con 0x7f2b440bfe70 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b427fc640 1 --2- 172.21.15.138:0/2143061470 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2b3003cef0 0x7f2b3003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b52ea6640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b52ea6640 10 monclient: _renew_subs 2024-09-11T05:07:53.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b52ea6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.882+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b440c4f40 con 0x7f2b440bfe70 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b52ea6640 1 librados: init done 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b427fc640 1 -- 172.21.15.138:0/2143061470 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f2b3c014da0 con 0x7f2b440bfe70 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b427fc640 10 client.4299.objecter ms_dispatch 0x7f2b4c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b427fc640 3 client.4299.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b427fc640 3 client.4299.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b427fc640 20 client.4299.objecter dump_active .. 0 homeless 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b52ea6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b52ea6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b52ea6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b51ea4640 1 --2- 172.21.15.138:0/2143061470 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2b3003cef0 0x7f2b3003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:53.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b44024af0 con 0x7f2b440bfe70 2024-09-11T05:07:53.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.886+0000 7f2b51ea4640 10 monclient: get_auth_request con 0x7f2b3003cef0 auth_method 0 2024-09-11T05:07:53.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.890+0000 7f2b427fc640 10 client.4299.objecter ms_handle_connect 0x7f2b3003cef0 2024-09-11T05:07:53.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.890+0000 7f2b51ea4640 1 --2- 172.21.15.138:0/2143061470 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2b3003cef0 0x7f2b3003f3b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2b48004470 tx=0x7f2b48003700 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:53.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.894+0000 7f2b427fc640 1 -- 172.21.15.138:0/2143061470 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f2b3c040030 con 0x7f2b440bfe70 2024-09-11T05:07:53.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.894+0000 7f2b427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:53.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.894+0000 7f2b427fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:53.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T05:07:53.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T05:07:53.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:53.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff034139640 con 0x7ff034070010 2024-09-11T05:07:53.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff0317fa640 1 -- 172.21.15.138:0/3745298604 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff02801bba0 con 0x7ff034070010 2024-09-11T05:07:53.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff0317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T05:07:53.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff0317fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:53.917 INFO:teuthology.orchestra.run.smithi138.stdout:47244640260 2024-09-11T05:07:53.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 10 librados: watch_flush enter 2024-09-11T05:07:53.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 10 librados: watch_flush exit 2024-09-11T05:07:53.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 20 client.4280.objecter shutdown clearing up homeless session... 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 10 client.4280.objecter successfully canceled tick 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff01803cef0 msgr2=0x7ff01803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 --2- 172.21.15.138:0/3745298604 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff01803cef0 0x7ff01803f3b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff0341398c0 tx=0x7ff020009040 comp rx=0 tx=0).stop 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 10 monclient: shutdown 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff034070010 msgr2=0x7ff0341d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff034070010 0x7ff0341d04f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff02800c470 tx=0x7ff02800c930 comp rx=0 tx=0).stop 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 shutdown_connections 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 --2- 172.21.15.138:0/3745298604 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff01803cef0 0x7ff01803f3b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff03409eb10 0x7ff0341408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff03409e6e0 0x7ff0341403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 --2- 172.21.15.138:0/3745298604 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff034070010 0x7ff0341d04f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 >> 172.21.15.138:0/3745298604 conn(0x7ff0340ab830 msgr2=0x7ff0340a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 shutdown_connections 2024-09-11T05:07:53.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 -- 172.21.15.138:0/3745298604 wait complete. 2024-09-11T05:07:53.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.898+0000 7ff03ad9f640 1 librados: shutdown 2024-09-11T05:07:53.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.906+0000 7fa2d2d96640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T05:07:53.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.906+0000 7fa2d2d96640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T05:07:53.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.906+0000 7fa2d2d96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.906+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa2cc06dd00 con 0x7fa2cc139c50 2024-09-11T05:07:53.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.906+0000 7fa2c97fa640 1 -- 172.21.15.138:0/2575279224 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa2c4019300 con 0x7fa2cc139c50 2024-09-11T05:07:53.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.906+0000 7fa2c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T05:07:53.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.906+0000 7fa2c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:53.921 INFO:teuthology.orchestra.run.smithi138.stdout:47244640261 2024-09-11T05:07:53.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 10 librados: watch_flush enter 2024-09-11T05:07:53.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 10 librados: watch_flush exit 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 20 client.4278.objecter shutdown clearing up homeless session... 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 10 client.4278.objecter successfully canceled tick 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa2b803cf00 msgr2=0x7fa2b803f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2575279224 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa2b803cf00 0x7fa2b803f3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009580 tx=0x7fa2c0008040 comp rx=0 tx=0).stop 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 10 monclient: shutdown 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2cc139c50 msgr2=0x7fa2cc1d11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2cc139c50 0x7fa2cc1d11b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa2c400caf0 tx=0x7fa2c400a060 comp rx=0 tx=0).stop 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 shutdown_connections 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2575279224 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa2b803cf00 0x7fa2b803f3c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa2cc139c50 0x7fa2cc1d11b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa2cc0a4260 0x7fa2cc1d0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 --2- 172.21.15.138:0/2575279224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2cc0a3940 0x7fa2cc1d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 >> 172.21.15.138:0/2575279224 conn(0x7fa2cc0a8530 msgr2=0x7fa2cc09fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 shutdown_connections 2024-09-11T05:07:53.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 -- 172.21.15.138:0/2575279224 wait complete. 2024-09-11T05:07:53.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.910+0000 7fa2d2d96640 1 librados: shutdown 2024-09-11T05:07:53.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.4 2024-09-11T05:07:53.925 DEBUG:teuthology.parallel:result is None 2024-09-11T05:07:53.937 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.0 2024-09-11T05:07:53.937 DEBUG:teuthology.parallel:result is None 2024-09-11T05:07:53.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.974+0000 7f1de06e8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T05:07:53.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.974+0000 7f1de06e8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T05:07:53.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.974+0000 7f1de06e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:53.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.974+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1dd8069fa0 con 0x7f1dd809eb10 2024-09-11T05:07:53.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.974+0000 7f1dceffd640 1 -- 172.21.15.138:0/3733837468 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1dd0012420 con 0x7f1dd809eb10 2024-09-11T05:07:53.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.974+0000 7f1dceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T05:07:53.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.974+0000 7f1dceffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:53.979 INFO:teuthology.orchestra.run.smithi138.stdout:47244640261 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 10 librados: watch_flush enter 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 10 librados: watch_flush exit 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 20 client.4287.objecter shutdown clearing up homeless session... 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 10 client.4287.objecter successfully canceled tick 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1dbc03cef0 msgr2=0x7f1dbc03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 --2- 172.21.15.138:0/3733837468 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1dbc03cef0 0x7f1dbc03f3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1dd40ac000 tx=0x7f1dd40a7070 comp rx=0 tx=0).stop 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 10 monclient: shutdown 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd809eb10 msgr2=0x7f1dd81d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 --2- 172.21.15.138:0/3733837468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd809eb10 0x7f1dd81d0ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c780 tx=0x7f1dd000cc40 comp rx=0 tx=0).stop 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 shutdown_connections 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 --2- 172.21.15.138:0/3733837468 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1dbc03cef0 0x7f1dbc03f3b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 --2- 172.21.15.138:0/3733837468 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1dd809eb10 0x7f1dd81d0ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 --2- 172.21.15.138:0/3733837468 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1dd809e6e0 0x7f1dd81d0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 --2- 172.21.15.138:0/3733837468 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1dd8070010 0x7f1dd81d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:53.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 >> 172.21.15.138:0/3733837468 conn(0x7f1dd80ab830 msgr2=0x7f1dd806ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:53.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 shutdown_connections 2024-09-11T05:07:53.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 -- 172.21.15.138:0/3733837468 wait complete. 2024-09-11T05:07:53.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:53.978+0000 7f1de06e8640 1 librados: shutdown 2024-09-11T05:07:53.995 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-09-11T05:07:53.996 DEBUG:teuthology.parallel:result is None 2024-09-11T05:07:54.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T05:07:54.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T05:07:54.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f17c40a4aa0 con 0x7f17c40a62a0 2024-09-11T05:07:54.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17c17fa640 1 -- 172.21.15.138:0/1162584785 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f17bc0165c0 con 0x7f17c40a62a0 2024-09-11T05:07:54.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T05:07:54.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17c17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:54.062 INFO:teuthology.orchestra.run.smithi138.stdout:51539607558 2024-09-11T05:07:54.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 10 librados: watch_flush enter 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 10 librados: watch_flush exit 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 20 client.4286.objecter shutdown clearing up homeless session... 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 10 client.4286.objecter successfully canceled tick 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 >> v2:172.21.15.190:6800/1562782189 conn(0x7f17a403cea0 msgr2=0x7f17a403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 --2- 172.21.15.138:0/1162584785 >> v2:172.21.15.190:6800/1562782189 conn(0x7f17a403cea0 0x7f17a403f360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f17b400aae0 tx=0x7f17b4008040 comp rx=0 tx=0).stop 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 10 monclient: shutdown 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a62a0 msgr2=0x7f17c413bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a62a0 0x7f17c413bda0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f17bc01d150 tx=0x7f17bc019650 comp rx=0 tx=0).stop 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 shutdown_connections 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 --2- 172.21.15.138:0/1162584785 >> v2:172.21.15.190:6800/1562782189 conn(0x7f17a403cea0 0x7f17a403f360 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17c40a62a0 0x7f17c413bda0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17c40a0850 0x7f17c4125f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 --2- 172.21.15.138:0/1162584785 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17c406ef10 0x7f17c4123fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 >> 172.21.15.138:0/1162584785 conn(0x7f17c40a9690 msgr2=0x7f17c409ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:54.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.058+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 shutdown_connections 2024-09-11T05:07:54.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7f17ca26e640 1 -- 172.21.15.138:0/1162584785 wait complete. 2024-09-11T05:07:54.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7f17ca26e640 1 librados: shutdown 2024-09-11T05:07:54.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7fd4cf336640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T05:07:54.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7fd4cf336640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T05:07:54.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7fd4cf336640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:54.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd4c8069fa0 con 0x7fd4c809e6e0 2024-09-11T05:07:54.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7fd4bdffb640 1 -- 172.21.15.138:0/2879720024 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd4b8012470 con 0x7fd4c809e6e0 2024-09-11T05:07:54.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7fd4bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T05:07:54.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.062+0000 7fd4bdffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:54.067 INFO:teuthology.orchestra.run.smithi138.stdout:47244640260 2024-09-11T05:07:54.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 10 librados: watch_flush enter 2024-09-11T05:07:54.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 10 librados: watch_flush exit 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 20 client.4293.objecter shutdown clearing up homeless session... 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 10 client.4293.objecter successfully canceled tick 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4ac03cef0 msgr2=0x7fd4ac03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 --2- 172.21.15.138:0/2879720024 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4ac03cef0 0x7fd4ac03f3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40ac000 tx=0x7fd4c40a7070 comp rx=0 tx=0).stop 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 10 monclient: shutdown 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c809e6e0 msgr2=0x7fd4c80a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 --2- 172.21.15.138:0/2879720024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c809e6e0 0x7fd4c80a0a60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c450 tx=0x7fd4b800c910 comp rx=0 tx=0).stop 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 shutdown_connections 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 --2- 172.21.15.138:0/2879720024 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4ac03cef0 0x7fd4ac03f3b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 --2- 172.21.15.138:0/2879720024 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4c809eb10 0x7fd4c80a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 --2- 172.21.15.138:0/2879720024 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4c809e6e0 0x7fd4c80a0a60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 --2- 172.21.15.138:0/2879720024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4c8070010 0x7fd4c80a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 >> 172.21.15.138:0/2879720024 conn(0x7fd4c80ab830 msgr2=0x7fd4c8137ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:54.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 shutdown_connections 2024-09-11T05:07:54.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 -- 172.21.15.138:0/2879720024 wait complete. 2024-09-11T05:07:54.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.066+0000 7fd4cf336640 1 librados: shutdown 2024-09-11T05:07:54.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.3 2024-09-11T05:07:54.081 DEBUG:teuthology.parallel:result is None 2024-09-11T05:07:54.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2024-09-11T05:07:54.087 DEBUG:teuthology.parallel:result is None 2024-09-11T05:07:54.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.134+0000 7f2b52ea6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T05:07:54.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.134+0000 7f2b52ea6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T05:07:54.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.134+0000 7f2b52ea6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:54.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.134+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f2b44008320 con 0x7f2b440bfe70 2024-09-11T05:07:54.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.134+0000 7f2b427fc640 1 -- 172.21.15.138:0/2143061470 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2b3c01dc00 con 0x7f2b440bfe70 2024-09-11T05:07:54.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.134+0000 7f2b427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T05:07:54.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.134+0000 7f2b427fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:54.140 INFO:teuthology.orchestra.run.smithi138.stdout:47244640260 2024-09-11T05:07:54.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 10 librados: watch_flush enter 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 10 librados: watch_flush exit 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 20 client.4299.objecter shutdown clearing up homeless session... 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 10 client.4299.objecter successfully canceled tick 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2b3003cef0 msgr2=0x7f2b3003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/2143061470 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2b3003cef0 0x7f2b3003f3b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2b48004470 tx=0x7f2b48003700 comp rx=0 tx=0).stop 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 10 monclient: shutdown 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440bfe70 msgr2=0x7f2b440be130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440bfe70 0x7f2b440be130 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00d620 tx=0x7f2b3c00dae0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 shutdown_connections 2024-09-11T05:07:54.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/2143061470 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2b3003cef0 0x7f2b3003f3b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b440c51a0 0x7f2b4401d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2b440bfe70 0x7f2b440be130 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 --2- 172.21.15.138:0/2143061470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2b440254c0 0x7f2b440bdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 >> 172.21.15.138:0/2143061470 conn(0x7f2b44006320 msgr2=0x7f2b44026320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:54.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 shutdown_connections 2024-09-11T05:07:54.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 -- 172.21.15.138:0/2143061470 wait complete. 2024-09-11T05:07:54.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.138+0000 7f2b52ea6640 1 librados: shutdown 2024-09-11T05:07:54.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.1 2024-09-11T05:07:54.156 DEBUG:teuthology.parallel:result is None 2024-09-11T05:07:54.156 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T05:07:54.157 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:54.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.258+0000 7f37a2479640 10 monclient: get_monmap_and_config 2024-09-11T05:07:54.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.258+0000 7f37a2479640 10 monclient: build_initial_monmap 2024-09-11T05:07:54.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.258+0000 7f37a2479640 10 monclient: monmap: 2024-09-11T05:07:54.263 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:54.263 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.263 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:54.264953+0000 2024-09-11T05:07:54.263 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:54.264953+0000 2024-09-11T05:07:54.264 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:54.264 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.264 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:54.264 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:54.264 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:54.264 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:54.264 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:54.264 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 1 Processor -- start 2024-09-11T05:07:54.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 1 -- start start 2024-09-11T05:07:54.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient: init 2024-09-11T05:07:54.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:54.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f379c0a5b40 0x7f379c0a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient(hunting): picked mon.noname-c con 0x7f379c0a5b40 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f379c0a7130 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c0a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient(hunting): picked mon.noname-b con 0x7f379c09d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f379c09d490 con 0x7f379c0a7130 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f379c09e110 con 0x7f379c0a5b40 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f379c0a69a0 con 0x7f379c09d840 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a2479640 10 monclient(hunting): authenticate will time out at 957.701046s 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c0a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c0a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58908/0 (socket says 172.21.15.138:58908) 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 1 -- 172.21.15.138:0/3952392921 learned_addr learned my addr 172.21.15.138:0/3952392921 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient(hunting): get_auth_request con 0x7f379c09d840 auth_method 0 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f379bfff640 1 --2- 172.21.15.138:0/3952392921 >> v2:172.21.15.190:3300/0 conn(0x7f379c0a5b40 0x7f379c0a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f379bfff640 10 monclient(hunting): get_auth_request con 0x7f379c0a5b40 auth_method 0 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f379bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f379bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f379bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-09-11T05:07:54.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 1 -- 172.21.15.138:0/3952392921 >> v2:172.21.15.190:3300/0 conn(0x7f379c0a5b40 msgr2=0x7f379c0a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:54.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 1 --2- 172.21.15.138:0/3952392921 >> v2:172.21.15.190:3300/0 conn(0x7f379c0a5b40 0x7f379c0a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 1 -- 172.21.15.138:0/3952392921 >> v1:172.21.15.190:6789/0 conn(0x7f379c0a7130 legacy=0x7f379c138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:54.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient: _finish_hunting 0 2024-09-11T05:07:54.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 1 monclient: found mon.noname-b 2024-09-11T05:07:54.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:54.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.262+0000 7f37a0a16640 1 -- 172.21.15.138:0/3952392921 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f379c070580 con 0x7f379c09d840 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a0a16640 1 -- 172.21.15.138:0/3952392921 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 msgr2=0x7f379c0a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a0a16640 1 --2- 172.21.15.138:0/3952392921 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c0a6460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3794009e50 tx=0x7f379401c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f2557fc3cd31d18 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 1 -- 172.21.15.138:0/3952392921 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940a5560 con 0x7f379c09d840 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: dump: 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.277 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 1 -- 172.21.15.138:0/3952392921 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37940a5700 con 0x7f379c09d840 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:54.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 1 -- 172.21.15.138:0/3952392921 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940a5ad0 con 0x7f379c09d840 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 5 monclient: authenticate success, global_id 4298 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 4 set_mon_vals no callback set 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f379b7fe640 10 monclient: dump: 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.279 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 10 monclient: shutdown 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 -- 172.21.15.138:0/3952392921 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 msgr2=0x7f379c0a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 --2- 172.21.15.138:0/3952392921 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c0a6460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3794009e50 tx=0x7f379401c8f0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 -- 172.21.15.138:0/3952392921 shutdown_connections 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 --2- 172.21.15.138:0/3952392921 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c0a6460 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 --2- 172.21.15.138:0/3952392921 >> v2:172.21.15.190:3300/0 conn(0x7f379c0a5b40 0x7f379c0a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 -- 172.21.15.138:0/3952392921 >> 172.21.15.138:0/3952392921 conn(0x7f379c0aba10 msgr2=0x7f379c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 -- 172.21.15.138:0/3952392921 shutdown_connections 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 -- 172.21.15.138:0/3952392921 wait complete. 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 10 monclient: build_initial_monmap 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 10 monclient: monmap: 2024-09-11T05:07:54.280 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 librados: starting msgr at 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.266+0000 7f37a2479640 1 librados: starting objecter 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 Processor -- start 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 -- start start 2024-09-11T05:07:54.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 librados: setting wanted keys 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 librados: calling monclient init 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient: init 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c13d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient(hunting): picked mon.noname-c con 0x7f379c09d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f379c0a5b40 0x7f379c13d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient(hunting): picked mon.noname-b con 0x7f379c0a5b40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f379c0a7130 0x7f379c13da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient(hunting): picked mon.noname-a con 0x7f379c0a7130 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f379c142f60 con 0x7f379c0a7130 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f379c142de0 con 0x7f379c0a5b40 2024-09-11T05:07:54.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f379c1430e0 con 0x7f379c09d840 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient(hunting): authenticate will time out at 957.708608s 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c13d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c13d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58910/0 (socket says 172.21.15.138:58910) 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 -- 172.21.15.138:0/352849157 learned_addr learned my addr 172.21.15.138:0/352849157 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient(hunting): get_auth_request con 0x7f379c09d840 auth_method 0 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f379c0a7130 0x7f379c13da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 10 monclient(hunting): get_auth_request con 0x7f379c0a7130 auth_method 0 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 -- 172.21.15.138:0/352849157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f379c0a5b40 msgr2=0x7f379c13d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f3793fff640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f379c0a5b40 0x7f379c13d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f379c0a5b40 0x7f379c13d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 -- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f379c0a7130 msgr2=0x7f379c13da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f379c0a7130 0x7f379c13da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 monclient: found mon.noname-c 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 -- 172.21.15.138:0/352849157 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f379c0a48a0 con 0x7f379c09d840 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f3793fff640 10 monclient: get_auth_request con 0x7f379c0a5b40 auth_method 0 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a0a16640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f379c0a7130 0x7f379c13da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f3793fff640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f379c0a5b40 0x7f379c13d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 -- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 msgr2=0x7f379c13d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379bfff640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c13d010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f378c00d800 tx=0x7f378c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 client.?.objecter ms_handle_connect 0x7f379c09d840 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 1 -- 172.21.15.138:0/352849157 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c018020 con 0x7f379c09d840 2024-09-11T05:07:54.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: dump: 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: _finish_auth 0 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: _check_auth_tickets 2024-09-11T05:07:54.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 1 -- 172.21.15.138:0/352849157 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f378c0098b0 con 0x7f379c09d840 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 1 -- 172.21.15.138:0/352849157 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c009c60 con 0x7f379c09d840 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37997fa640 10 monclient: dump: 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.287 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 5 monclient: authenticate success, global_id 4301 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f379affd640 4 set_mon_vals no callback set 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient: _renew_subs 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f379c121be0 con 0x7f379c09d840 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient: _renew_subs 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f379c1224b0 con 0x7f379c09d840 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.270+0000 7f37a2479640 1 librados: init done 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37a2479640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37a2479640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37a2479640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f379c138900 con 0x7f379c09d840 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37997fa640 1 -- 172.21.15.138:0/352849157 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f378c013050 con 0x7f379c09d840 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37997fa640 1 --2- 172.21.15.138:0/352849157 >> v2:172.21.15.190:6800/1562782189 conn(0x7f377c03cef0 0x7f377c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37997fa640 1 -- 172.21.15.138:0/352849157 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f378c04f810 con 0x7f379c09d840 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37997fa640 10 client.4301.objecter ms_dispatch 0x7f379c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37997fa640 3 client.4301.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37997fa640 3 client.4301.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f37997fa640 20 client.4301.objecter dump_active .. 0 homeless 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f3793fff640 1 --2- 172.21.15.138:0/352849157 >> v2:172.21.15.190:6800/1562782189 conn(0x7f377c03cef0 0x7f377c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.274+0000 7f3793fff640 10 monclient: get_auth_request con 0x7f377c03cef0 auth_method 0 2024-09-11T05:07:54.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.278+0000 7f37997fa640 10 client.4301.objecter ms_handle_connect 0x7f377c03cef0 2024-09-11T05:07:54.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.278+0000 7f3793fff640 1 --2- 172.21.15.138:0/352849157 >> v2:172.21.15.190:6800/1562782189 conn(0x7f377c03cef0 0x7f377c03f3b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f379c1388d0 tx=0x7f3784009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.278+0000 7f37997fa640 1 -- 172.21.15.138:0/352849157 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f378c022030 con 0x7f379c09d840 2024-09-11T05:07:54.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.278+0000 7f37997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.278+0000 7f37997fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:54.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.430+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f379c069fa0 con 0x7f377c03cef0 2024-09-11T05:07:54.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37997fa640 1 -- 172.21.15.138:0/352849157 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34177 (secure 0 0 0) 0x7f379c069fa0 con 0x7f377c03cef0 2024-09-11T05:07:54.436 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:54.436 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:07:54.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 10 librados: watch_flush enter 2024-09-11T05:07:54.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 10 librados: watch_flush exit 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 20 client.4301.objecter shutdown clearing up homeless session... 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 10 client.4301.objecter successfully canceled tick 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 >> v2:172.21.15.190:6800/1562782189 conn(0x7f377c03cef0 msgr2=0x7f377c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 --2- 172.21.15.138:0/352849157 >> v2:172.21.15.190:6800/1562782189 conn(0x7f377c03cef0 0x7f377c03f3b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f379c1388d0 tx=0x7f3784009040 comp rx=0 tx=0).stop 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 10 monclient: shutdown 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 msgr2=0x7f379c13d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c13d010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f378c00d800 tx=0x7f378c00dcc0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 shutdown_connections 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 --2- 172.21.15.138:0/352849157 >> v2:172.21.15.190:6800/1562782189 conn(0x7f377c03cef0 0x7f377c03f3b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f379c0a7130 0x7f379c13da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f379c0a5b40 0x7f379c13d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 --2- 172.21.15.138:0/352849157 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f379c09d840 0x7f379c13d010 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 >> 172.21.15.138:0/352849157 conn(0x7f379c0aba10 msgr2=0x7f379c0a4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:54.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 shutdown_connections 2024-09-11T05:07:54.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 -- 172.21.15.138:0/352849157 wait complete. 2024-09-11T05:07:54.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.434+0000 7f37a2479640 1 librados: shutdown 2024-09-11T05:07:54.453 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-11T05:07:52.665078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163284,"kb_used_data":1080,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067756,"statfs":{"total":579820584960,"available":579653382144,"internally_reserved":0,"allocated":1105920,"data_stored":1370890,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.149110"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.536048+0000","last_change":"2024-09-11T05:07:50.536048+0000","last_active":"2024-09-11T05:07:50.536048+0000","last_peered":"2024-09-11T05:07:50.536048+0000","last_clean":"2024-09-11T05:07:50.536048+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:07:50.536048+0000","last_undegraded":"2024-09-11T05:07:50.536048+0000","last_fullsized":"2024-09-11T05:07:50.536048+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:33.527347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543840+0000","last_change":"2024-09-11T05:07:50.543840+0000","last_active":"2024-09-11T05:07:50.543840+0000","last_peered":"2024-09-11T05:07:50.543840+0000","last_clean":"2024-09-11T05:07:50.543840+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:07:50.543840+0000","last_undegraded":"2024-09-11T05:07:50.543840+0000","last_fullsized":"2024-09-11T05:07:50.543840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:09.809820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.535722+0000","last_change":"2024-09-11T05:07:50.535722+0000","last_active":"2024-09-11T05:07:50.535722+0000","last_peered":"2024-09-11T05:07:50.535722+0000","last_clean":"2024-09-11T05:07:50.535722+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:07:50.535722+0000","last_undegraded":"2024-09-11T05:07:50.535722+0000","last_fullsized":"2024-09-11T05:07:50.535722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:07.310720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543876+0000","last_change":"2024-09-11T05:07:50.543876+0000","last_active":"2024-09-11T05:07:50.543876+0000","last_peered":"2024-09-11T05:07:50.543876+0000","last_clean":"2024-09-11T05:07:50.543876+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:07:50.543876+0000","last_undegraded":"2024-09-11T05:07:50.543876+0000","last_fullsized":"2024-09-11T05:07:50.543876+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:47:12.140333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543322+0000","last_change":"2024-09-11T05:07:50.541650+0000","last_active":"2024-09-11T05:07:50.543322+0000","last_peered":"2024-09-11T05:07:50.543322+0000","last_clean":"2024-09-11T05:07:50.543322+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:07:50.543322+0000","last_undegraded":"2024-09-11T05:07:50.543322+0000","last_fullsized":"2024-09-11T05:07:50.543322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:47.451082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.533338+0000","last_change":"2024-09-11T05:07:50.533338+0000","last_active":"2024-09-11T05:07:50.533338+0000","last_peered":"2024-09-11T05:07:50.533338+0000","last_clean":"2024-09-11T05:07:50.533338+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:07:50.533338+0000","last_undegraded":"2024-09-11T05:07:50.533338+0000","last_fullsized":"2024-09-11T05:07:50.533338+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:04.573351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.535448+0000","last_change":"2024-09-11T05:07:50.535448+0000","last_active":"2024-09-11T05:07:50.535448+0000","last_peered":"2024-09-11T05:07:50.535448+0000","last_clean":"2024-09-11T05:07:50.535448+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:07:50.535448+0000","last_undegraded":"2024-09-11T05:07:50.535448+0000","last_fullsized":"2024-09-11T05:07:50.535448+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:57.570691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.540810+0000","last_change":"2024-09-11T05:07:50.540810+0000","last_active":"2024-09-11T05:07:50.540810+0000","last_peered":"2024-09-11T05:07:50.540810+0000","last_clean":"2024-09-11T05:07:50.540810+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:07:50.540810+0000","last_undegraded":"2024-09-11T05:07:50.540810+0000","last_fullsized":"2024-09-11T05:07:50.540810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:39.133232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.534426+0000","last_change":"2024-09-11T05:07:43.495116+0000","last_active":"2024-09-11T05:07:50.534426+0000","last_peered":"2024-09-11T05:07:50.534426+0000","last_clean":"2024-09-11T05:07:50.534426+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:07:50.534426+0000","last_undegraded":"2024-09-11T05:07:50.534426+0000","last_fullsized":"2024-09-11T05:07:50.534426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_clean_scrub_stamp":"2024-09-11T05:07:41.703163+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:01.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":581782,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":48726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":581782,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:54.454 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:54.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.554+0000 7f1ae8276640 10 monclient: get_monmap_and_config 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.554+0000 7f1ae8276640 10 monclient: build_initial_monmap 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.554+0000 7f1ae8276640 10 monclient: monmap: 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:54.560463+0000 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:54.560463+0000 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:54.559 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:54.560 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:54.560 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:54.560 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 1 Processor -- start 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 1 -- start start 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient: init 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1ae006e0e0 0x7f1ae009e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient(hunting): picked mon.noname-c con 0x7f1ae006e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1ae009e960 0x7f1ae009ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient(hunting): picked mon.noname-a con 0x7f1ae009e960 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:54.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ae01356f0 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae006e4c0 con 0x7f1ae01356f0 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae00a19b0 con 0x7f1ae009e960 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae009f820 con 0x7f1ae006e0e0 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae8276640 10 monclient(hunting): authenticate will time out at 957.997013s 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1ae009e960 0x7f1ae009ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1ae009e960 0x7f1ae009ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46508/0 (socket says 172.21.15.138:46508) 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 1 -- 172.21.15.138:0/1867507722 learned_addr learned my addr 172.21.15.138:0/1867507722 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient(hunting): get_auth_request con 0x7f1ae009e960 auth_method 0 2024-09-11T05:07:54.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 1 -- 172.21.15.138:0/1867507722 >> v2:172.21.15.190:3300/0 conn(0x7f1ae006e0e0 msgr2=0x7f1ae009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae6012640 1 --2- 172.21.15.138:0/1867507722 >> v2:172.21.15.190:3300/0 conn(0x7f1ae006e0e0 0x7f1ae009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 1 --2- 172.21.15.138:0/1867507722 >> v2:172.21.15.190:3300/0 conn(0x7f1ae006e0e0 0x7f1ae009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 1 -- 172.21.15.138:0/1867507722 >> v1:172.21.15.138:6789/0 conn(0x7f1ae01356f0 legacy=0x7f1ae0135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient: _finish_hunting 0 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 1 monclient: found mon.noname-a 2024-09-11T05:07:54.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:54.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:54.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.558+0000 7f1ae5811640 1 -- 172.21.15.138:0/1867507722 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae009f540 con 0x7f1ae009e960 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5811640 1 -- 172.21.15.138:0/1867507722 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae009e960 msgr2=0x7f1ae009ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5811640 1 --2- 172.21.15.138:0/1867507722 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae009e960 0x7f1ae009ed70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0069420 tx=0x7f1ad001eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818a9fd761e25bc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 1 -- 172.21.15.138:0/1867507722 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00b4070 con 0x7f1ae009e960 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: dump: 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.567 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: _finish_auth 0 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: _check_auth_tickets 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 5 monclient: authenticate success, global_id 4305 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 1 -- 172.21.15.138:0/1867507722 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad00a79b0 con 0x7f1ae009e960 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:54.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 4 set_mon_vals no callback set 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 1 -- 172.21.15.138:0/1867507722 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00a7d80 con 0x7f1ae009e960 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae5010640 10 monclient: dump: 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 10 monclient: shutdown 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 -- 172.21.15.138:0/1867507722 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae009e960 msgr2=0x7f1ae009ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 --2- 172.21.15.138:0/1867507722 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae009e960 0x7f1ae009ed70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0069420 tx=0x7f1ad001eab0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 -- 172.21.15.138:0/1867507722 shutdown_connections 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 --2- 172.21.15.138:0/1867507722 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae009e960 0x7f1ae009ed70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 --2- 172.21.15.138:0/1867507722 >> v2:172.21.15.190:3300/0 conn(0x7f1ae006e0e0 0x7f1ae009e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 -- 172.21.15.138:0/1867507722 >> 172.21.15.138:0/1867507722 conn(0x7f1ae0063640 msgr2=0x7f1ae0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 -- 172.21.15.138:0/1867507722 shutdown_connections 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 -- 172.21.15.138:0/1867507722 wait complete. 2024-09-11T05:07:54.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 10 monclient: build_initial_monmap 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 10 monclient: monmap: 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 librados: starting msgr at 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 librados: starting objecter 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 Processor -- start 2024-09-11T05:07:54.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 -- start start 2024-09-11T05:07:54.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 librados: setting wanted keys 2024-09-11T05:07:54.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 1 librados: calling monclient init 2024-09-11T05:07:54.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.562+0000 7f1ae8276640 10 monclient: init 2024-09-11T05:07:54.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:54.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae006e0e0 0x7f1ae01cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient(hunting): picked mon.noname-b con 0x7f1ae006e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1ae009e960 0x7f1ae0134820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient(hunting): picked mon.noname-c con 0x7f1ae009e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae01356f0 0x7f1ae013b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient(hunting): picked mon.noname-a con 0x7f1ae01356f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae00a2b40 con 0x7f1ae01356f0 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae00a29c0 con 0x7f1ae006e0e0 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae00a2cc0 con 0x7f1ae009e960 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient(hunting): authenticate will time out at 958.004175s 2024-09-11T05:07:54.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1ae009e960 0x7f1ae0134820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae01356f0 0x7f1ae013b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1ae009e960 0x7f1ae0134820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58912/0 (socket says 172.21.15.138:58912) 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 1 -- 172.21.15.138:0/3638449949 learned_addr learned my addr 172.21.15.138:0/3638449949 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 10 monclient(hunting): get_auth_request con 0x7f1ae009e960 auth_method 0 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient(hunting): get_auth_request con 0x7f1ae01356f0 auth_method 0 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6012640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae006e0e0 0x7f1ae01cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 -- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1ae009e960 msgr2=0x7f1ae0134820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1ae009e960 0x7f1ae0134820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 -- 172.21.15.138:0/3638449949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae006e0e0 msgr2=0x7f1ae01cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae006e0e0 0x7f1ae01cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient: _finish_hunting 0 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 monclient: found mon.noname-a 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:54.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 -- 172.21.15.138:0/3638449949 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae013d6d0 con 0x7f1ae01356f0 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6012640 10 monclient: get_auth_request con 0x7f1ae006e0e0 auth_method 0 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae5811640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1ae009e960 0x7f1ae0134820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6012640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae006e0e0 0x7f1ae01cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 -- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae01356f0 msgr2=0x7f1ae013b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae6813640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae01356f0 0x7f1ae013b7c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8004320 tx=0x7f1ad80047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 client.?.objecter ms_handle_connect 0x7f1ae01356f0 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 1 -- 172.21.15.138:0/3638449949 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8010900 con 0x7f1ae01356f0 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: dump: 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.579 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 1 -- 172.21.15.138:0/3638449949 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad8010aa0 con 0x7f1ae01356f0 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:54.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 1 -- 172.21.15.138:0/3638449949 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad800c450 con 0x7f1ae01356f0 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ad6ffd640 10 monclient: dump: 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.581 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae480f640 4 set_mon_vals no callback set 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 5 monclient: authenticate success, global_id 4308 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient: _renew_subs 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae011e910 con 0x7f1ae01356f0 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient: _renew_subs 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae011eed0 con 0x7f1ae01356f0 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.566+0000 7f1ae8276640 1 librados: init done 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ad6ffd640 1 -- 172.21.15.138:0/3638449949 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1ad8016070 con 0x7f1ae01356f0 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ae8276640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ae8276640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ae8276640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:54.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae0135520 con 0x7f1ae01356f0 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ad6ffd640 1 --2- 172.21.15.138:0/3638449949 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1ac403cf20 0x7f1ac403f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ad6ffd640 1 -- 172.21.15.138:0/3638449949 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f1ad8052f90 con 0x7f1ae01356f0 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ad6ffd640 10 client.4308.objecter ms_dispatch 0x7f1ae0000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ad6ffd640 3 client.4308.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ad6ffd640 3 client.4308.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ad6ffd640 20 client.4308.objecter dump_active .. 0 homeless 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ae6012640 1 --2- 172.21.15.138:0/3638449949 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1ac403cf20 0x7f1ac403f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ae6012640 10 monclient: get_auth_request con 0x7f1ac403cf20 auth_method 0 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ad6ffd640 10 client.4308.objecter ms_handle_connect 0x7f1ac403cf20 2024-09-11T05:07:54.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.570+0000 7f1ae6012640 1 --2- 172.21.15.138:0/3638449949 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1ac403cf20 0x7f1ac403f3e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1adc006fd0 tx=0x7f1adc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.574+0000 7f1ad6ffd640 1 -- 172.21.15.138:0/3638449949 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f1ad8051e50 con 0x7f1ae01356f0 2024-09-11T05:07:54.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.574+0000 7f1ad6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.574+0000 7f1ad6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:54.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.726+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ae009e420 con 0x7f1ac403cf20 2024-09-11T05:07:54.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ad6ffd640 1 -- 172.21.15.138:0/3638449949 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34177 (secure 0 0 0) 0x7f1ae009e420 con 0x7f1ac403cf20 2024-09-11T05:07:54.733 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:54.733 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:07:54.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 10 librados: watch_flush enter 2024-09-11T05:07:54.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 10 librados: watch_flush exit 2024-09-11T05:07:54.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 20 client.4308.objecter shutdown clearing up homeless session... 2024-09-11T05:07:54.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 10 client.4308.objecter successfully canceled tick 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1ac403cf20 msgr2=0x7f1ac403f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 --2- 172.21.15.138:0/3638449949 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1ac403cf20 0x7f1ac403f3e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1adc006fd0 tx=0x7f1adc008040 comp rx=0 tx=0).stop 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 10 monclient: shutdown 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae01356f0 msgr2=0x7f1ae013b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae01356f0 0x7f1ae013b7c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8004320 tx=0x7f1ad80047e0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 shutdown_connections 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 --2- 172.21.15.138:0/3638449949 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1ac403cf20 0x7f1ac403f3e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1ae01356f0 0x7f1ae013b7c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1ae009e960 0x7f1ae0134820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 --2- 172.21.15.138:0/3638449949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae006e0e0 0x7f1ae01cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 >> 172.21.15.138:0/3638449949 conn(0x7f1ae0063640 msgr2=0x7f1ae00a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:54.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.730+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 shutdown_connections 2024-09-11T05:07:54.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.734+0000 7f1ae8276640 1 -- 172.21.15.138:0/3638449949 wait complete. 2024-09-11T05:07:54.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.734+0000 7f1ae8276640 1 librados: shutdown 2024-09-11T05:07:54.750 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-11T05:07:54.665699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163284,"kb_used_data":1080,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067756,"statfs":{"total":579820584960,"available":579653382144,"internally_reserved":0,"allocated":1105920,"data_stored":1370890,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.144925"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.536048+0000","last_change":"2024-09-11T05:07:50.536048+0000","last_active":"2024-09-11T05:07:50.536048+0000","last_peered":"2024-09-11T05:07:50.536048+0000","last_clean":"2024-09-11T05:07:50.536048+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:07:50.536048+0000","last_undegraded":"2024-09-11T05:07:50.536048+0000","last_fullsized":"2024-09-11T05:07:50.536048+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:33.527347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543840+0000","last_change":"2024-09-11T05:07:50.543840+0000","last_active":"2024-09-11T05:07:50.543840+0000","last_peered":"2024-09-11T05:07:50.543840+0000","last_clean":"2024-09-11T05:07:50.543840+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:07:50.543840+0000","last_undegraded":"2024-09-11T05:07:50.543840+0000","last_fullsized":"2024-09-11T05:07:50.543840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:09.809820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.535722+0000","last_change":"2024-09-11T05:07:50.535722+0000","last_active":"2024-09-11T05:07:50.535722+0000","last_peered":"2024-09-11T05:07:50.535722+0000","last_clean":"2024-09-11T05:07:50.535722+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:07:50.535722+0000","last_undegraded":"2024-09-11T05:07:50.535722+0000","last_fullsized":"2024-09-11T05:07:50.535722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:07.310720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543876+0000","last_change":"2024-09-11T05:07:50.543876+0000","last_active":"2024-09-11T05:07:50.543876+0000","last_peered":"2024-09-11T05:07:50.543876+0000","last_clean":"2024-09-11T05:07:50.543876+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:07:50.543876+0000","last_undegraded":"2024-09-11T05:07:50.543876+0000","last_fullsized":"2024-09-11T05:07:50.543876+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:47:12.140333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543322+0000","last_change":"2024-09-11T05:07:50.541650+0000","last_active":"2024-09-11T05:07:50.543322+0000","last_peered":"2024-09-11T05:07:50.543322+0000","last_clean":"2024-09-11T05:07:50.543322+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:07:50.543322+0000","last_undegraded":"2024-09-11T05:07:50.543322+0000","last_fullsized":"2024-09-11T05:07:50.543322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:47.451082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.533338+0000","last_change":"2024-09-11T05:07:50.533338+0000","last_active":"2024-09-11T05:07:50.533338+0000","last_peered":"2024-09-11T05:07:50.533338+0000","last_clean":"2024-09-11T05:07:50.533338+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:07:50.533338+0000","last_undegraded":"2024-09-11T05:07:50.533338+0000","last_fullsized":"2024-09-11T05:07:50.533338+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:04.573351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.535448+0000","last_change":"2024-09-11T05:07:50.535448+0000","last_active":"2024-09-11T05:07:50.535448+0000","last_peered":"2024-09-11T05:07:50.535448+0000","last_clean":"2024-09-11T05:07:50.535448+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:07:50.535448+0000","last_undegraded":"2024-09-11T05:07:50.535448+0000","last_fullsized":"2024-09-11T05:07:50.535448+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:57.570691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.540810+0000","last_change":"2024-09-11T05:07:50.540810+0000","last_active":"2024-09-11T05:07:50.540810+0000","last_peered":"2024-09-11T05:07:50.540810+0000","last_clean":"2024-09-11T05:07:50.540810+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:07:50.540810+0000","last_undegraded":"2024-09-11T05:07:50.540810+0000","last_fullsized":"2024-09-11T05:07:50.540810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:39.133232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.534426+0000","last_change":"2024-09-11T05:07:43.495116+0000","last_active":"2024-09-11T05:07:50.534426+0000","last_peered":"2024-09-11T05:07:50.534426+0000","last_clean":"2024-09-11T05:07:50.534426+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:07:50.534426+0000","last_undegraded":"2024-09-11T05:07:50.534426+0000","last_fullsized":"2024-09-11T05:07:50.534426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_clean_scrub_stamp":"2024-09-11T05:07:41.703163+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:01.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":581782,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":48726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":581782,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:54.751 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T05:07:54.751 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T05:07:54.751 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T05:07:54.751 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T05:07:54.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient: get_monmap_and_config 2024-09-11T05:07:54.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient: build_initial_monmap 2024-09-11T05:07:54.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient: monmap: 2024-09-11T05:07:54.856 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:54.858345+0000 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:54.858345+0000 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:54.857 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 1 Processor -- start 2024-09-11T05:07:54.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 1 -- start start 2024-09-11T05:07:54.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient: init 2024-09-11T05:07:54.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07300a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient(hunting): picked mon.noname-b con 0x7f07300a5b40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07300a7130 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f073009d840 0x7f07300a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient(hunting): picked mon.noname-a con 0x7f073009d840 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f073009d490 con 0x7f07300a7130 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f073009e110 con 0x7f073009d840 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f07300a69a0 con 0x7f07300a5b40 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:54.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f07365d0640 10 monclient(hunting): authenticate will time out at 958.294330s 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f073009d840 0x7f07300a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07300a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f073009d840 0x7f07300a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46522/0 (socket says 172.21.15.138:46522) 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 1 -- 172.21.15.138:0/1452835069 learned_addr learned my addr 172.21.15.138:0/1452835069 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 10 monclient(hunting): get_auth_request con 0x7f073009d840 auth_method 0 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient(hunting): get_auth_request con 0x7f07300a5b40 auth_method 0 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 1 -- 172.21.15.138:0/1452835069 >> v2:172.21.15.138:3300/0 conn(0x7f073009d840 msgr2=0x7f07300a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 1 --2- 172.21.15.138:0/1452835069 >> v2:172.21.15.138:3300/0 conn(0x7f073009d840 0x7f07300a6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 1 -- 172.21.15.138:0/1452835069 >> v1:172.21.15.190:6789/0 conn(0x7f07300a7130 legacy=0x7f0730138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 1 monclient: found mon.noname-b 2024-09-11T05:07:54.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:54.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f072ffff640 1 -- 172.21.15.138:0/1452835069 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0730070580 con 0x7f07300a5b40 2024-09-11T05:07:54.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.854+0000 7f0734b6d640 1 --2- 172.21.15.138:0/1452835069 >> v2:172.21.15.138:3300/0 conn(0x7f073009d840 0x7f07300a6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:54.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072ffff640 1 -- 172.21.15.138:0/1452835069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 msgr2=0x7f07300a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:54.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072ffff640 1 --2- 172.21.15.138:0/1452835069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07300a5f20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f07200097c0 tx=0x7f072001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0e65ea8850a58da server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 1 -- 172.21.15.138:0/1452835069 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07200b0070 con 0x7f07300a5b40 2024-09-11T05:07:54.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: dump: 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: _finish_auth 0 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 1 -- 172.21.15.138:0/1452835069 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07200ab600 con 0x7f07300a5b40 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 5 monclient: authenticate success, global_id 4307 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 4 set_mon_vals no callback set 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 1 -- 172.21.15.138:0/1452835069 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07200ab9d0 con 0x7f07300a5b40 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f072effd640 10 monclient: dump: 2024-09-11T05:07:54.868 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 10 monclient: shutdown 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 1 -- 172.21.15.138:0/1452835069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 msgr2=0x7f07300a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 1 --2- 172.21.15.138:0/1452835069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07300a5f20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f07200097c0 tx=0x7f072001c360 comp rx=0 tx=0).stop 2024-09-11T05:07:54.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 1 -- 172.21.15.138:0/1452835069 shutdown_connections 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 1 --2- 172.21.15.138:0/1452835069 >> v2:172.21.15.138:3300/0 conn(0x7f073009d840 0x7f07300a6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 1 --2- 172.21.15.138:0/1452835069 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07300a5f20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 1 -- 172.21.15.138:0/1452835069 >> 172.21.15.138:0/1452835069 conn(0x7f07300aba10 msgr2=0x7f07300a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 1 -- 172.21.15.138:0/1452835069 shutdown_connections 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.858+0000 7f07365d0640 1 -- 172.21.15.138:0/1452835069 wait complete. 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient: build_initial_monmap 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient: monmap: 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.870 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 librados: starting msgr at 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 librados: starting objecter 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 Processor -- start 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 -- start start 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 librados: setting wanted keys 2024-09-11T05:07:54.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 librados: calling monclient init 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient: init 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073009d840 0x7f07301d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient(hunting): picked mon.noname-b con 0x7f073009d840 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07301379f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient(hunting): picked mon.noname-c con 0x7f07300a5b40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07300a7130 0x7f073013da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient(hunting): picked mon.noname-a con 0x7f07300a7130 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0730142f00 con 0x7f07300a7130 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0730142d80 con 0x7f073009d840 2024-09-11T05:07:54.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0730143080 con 0x7f07300a5b40 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 10 monclient(hunting): authenticate will time out at 958.301185s 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07301379f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f0734b6d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07300a7130 0x7f073013da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07301379f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58932/0 (socket says 172.21.15.138:58932) 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 -- 172.21.15.138:0/3977000363 learned_addr learned my addr 172.21.15.138:0/3977000363 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient(hunting): get_auth_request con 0x7f07300a5b40 auth_method 0 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f0734b6d640 10 monclient(hunting): get_auth_request con 0x7f07300a7130 auth_method 0 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f0734b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f0734b6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:54.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f0734b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f0734b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f0734b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f0734b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 -- 172.21.15.138:0/3977000363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073009d840 msgr2=0x7f07301d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 --2- 172.21.15.138:0/3977000363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073009d840 0x7f07301d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 -- 172.21.15.138:0/3977000363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07300a7130 msgr2=0x7f073013da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 --2- 172.21.15.138:0/3977000363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07300a7130 0x7f073013da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 monclient: found mon.noname-c 2024-09-11T05:07:54.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:54.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 -- 172.21.15.138:0/3977000363 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0730121dc0 con 0x7f07300a5b40 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 -- 172.21.15.138:0/3977000363 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 msgr2=0x7f07301379f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072f7fe640 1 --2- 172.21.15.138:0/3977000363 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07301379f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f072400c450 tx=0x7f072400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 10 client.?.objecter ms_handle_connect 0x7f07300a5b40 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 1 -- 172.21.15.138:0/3977000363 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724012a40 con 0x7f07300a5b40 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 10 monclient: dump: 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.877 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 10 monclient: _finish_auth 0 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 10 monclient: _check_auth_tickets 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f072cff9640 1 -- 172.21.15.138:0/3977000363 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0724017070 con 0x7f07300a5b40 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.862+0000 7f07365d0640 5 monclient: authenticate success, global_id 4310 2024-09-11T05:07:54.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 10 monclient: _renew_subs 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07301220a0 con 0x7f07300a5b40 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 1 -- 172.21.15.138:0/3977000363 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724012e40 con 0x7f07300a5b40 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 10 monclient: dump: 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:54.879 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072e7fc640 4 set_mon_vals no callback set 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 10 monclient: _renew_subs 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07301d11f0 con 0x7f07300a5b40 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 1 librados: init done 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0730138900 con 0x7f07300a5b40 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 1 -- 172.21.15.138:0/3977000363 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f072401c460 con 0x7f07300a5b40 2024-09-11T05:07:54.880 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 1 --2- 172.21.15.138:0/3977000363 >> v2:172.21.15.190:6800/1562782189 conn(0x7f071c03cef0 0x7f071c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 1 -- 172.21.15.138:0/3977000363 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f072404f070 con 0x7f07300a5b40 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 10 client.4310.objecter ms_dispatch 0x7f0730000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 3 client.4310.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 3 client.4310.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072cff9640 20 client.4310.objecter dump_active .. 0 homeless 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072ffff640 1 --2- 172.21.15.138:0/3977000363 >> v2:172.21.15.190:6800/1562782189 conn(0x7f071c03cef0 0x7f071c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.866+0000 7f072ffff640 10 monclient: get_auth_request con 0x7f071c03cef0 auth_method 0 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.870+0000 7f072cff9640 10 client.4310.objecter ms_handle_connect 0x7f071c03cef0 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.870+0000 7f072ffff640 1 --2- 172.21.15.138:0/3977000363 >> v2:172.21.15.190:6800/1562782189 conn(0x7f071c03cef0 0x7f071c03f3b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f073006e8d0 tx=0x7f0720004740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.870+0000 7f072cff9640 1 -- 172.21.15.138:0/3977000363 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f072401ac60 con 0x7f07300a5b40 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.870+0000 7f072cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:54.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:54.870+0000 7f072cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:55.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.058+0000 7f07365d0640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-11T05:07:55.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.058+0000 7f07365d0640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-11T05:07:55.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.058+0000 7f07365d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.058+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f07300aa9d0 con 0x7f07300a5b40 2024-09-11T05:07:55.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f072cff9640 1 -- 172.21.15.138:0/3977000363 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f072401c960 con 0x7f07300a5b40 2024-09-11T05:07:55.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f072cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-11T05:07:55.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f072cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:55.064 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:55.064 INFO:teuthology.orchestra.run.smithi138.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T05:07:55.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 10 librados: watch_flush enter 2024-09-11T05:07:55.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 10 librados: watch_flush exit 2024-09-11T05:07:55.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 20 client.4310.objecter shutdown clearing up homeless session... 2024-09-11T05:07:55.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 10 client.4310.objecter successfully canceled tick 2024-09-11T05:07:55.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 >> v2:172.21.15.190:6800/1562782189 conn(0x7f071c03cef0 msgr2=0x7f071c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 --2- 172.21.15.138:0/3977000363 >> v2:172.21.15.190:6800/1562782189 conn(0x7f071c03cef0 0x7f071c03f3b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f073006e8d0 tx=0x7f0720004740 comp rx=0 tx=0).stop 2024-09-11T05:07:55.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 10 monclient: shutdown 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 msgr2=0x7f07301379f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 --2- 172.21.15.138:0/3977000363 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07301379f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f072400c450 tx=0x7f072400c910 comp rx=0 tx=0).stop 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 shutdown_connections 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 --2- 172.21.15.138:0/3977000363 >> v2:172.21.15.190:6800/1562782189 conn(0x7f071c03cef0 0x7f071c03f3b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 --2- 172.21.15.138:0/3977000363 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07300a7130 0x7f073013da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 --2- 172.21.15.138:0/3977000363 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07300a5b40 0x7f07301379f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 --2- 172.21.15.138:0/3977000363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f073009d840 0x7f07301d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 >> 172.21.15.138:0/3977000363 conn(0x7f07300aba10 msgr2=0x7f07300a46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 shutdown_connections 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 -- 172.21.15.138:0/3977000363 wait complete. 2024-09-11T05:07:55.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.062+0000 7f07365d0640 1 librados: shutdown 2024-09-11T05:07:55.081 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T05:07:55.081 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-11T05:07:55.094 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-11T05:07:55.094 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:07:55.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient: get_monmap_and_config 2024-09-11T05:07:55.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient: build_initial_monmap 2024-09-11T05:07:55.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient: monmap: 2024-09-11T05:07:55.209 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:55.209 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.209 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:55.210981+0000 2024-09-11T05:07:55.209 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:55.210981+0000 2024-09-11T05:07:55.210 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:55.210 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.210 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:55.210 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:55.210 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:55.210 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:55.210 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:55.210 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 1 Processor -- start 2024-09-11T05:07:55.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 1 -- start start 2024-09-11T05:07:55.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient: init 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f17280a5b40 0x7f17280a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient(hunting): picked mon.noname-c con 0x7f17280a5b40 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17280a7130 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f172809d840 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f172809d490 con 0x7f172809d840 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1728139720 con 0x7f17280a7130 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f172809e110 con 0x7f17280a5b40 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f172dfa7640 10 monclient(hunting): authenticate will time out at 958.647779s 2024-09-11T05:07:55.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f1727fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f172809d840 0x7f1728139010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38454/0 (socket says 172.21.15.138:38454) 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.206+0000 7f1727fff640 1 -- 172.21.15.138:0/3678555772 learned_addr learned my addr 172.21.15.138:0/3678555772 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 1 --2- 172.21.15.138:0/3678555772 >> v2:172.21.15.190:3300/0 conn(0x7f17280a5b40 0x7f17280a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient(hunting): get_auth_request con 0x7f17280a5b40 auth_method 0 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 1 -- 172.21.15.138:0/3678555772 >> v1:172.21.15.190:6789/0 conn(0x7f17280a7130 legacy=0x7f1728138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 1 -- 172.21.15.138:0/3678555772 >> v1:172.21.15.138:6789/0 conn(0x7f172809d840 legacy=0x7f1728139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:55.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 1 monclient: found mon.noname-c 2024-09-11T05:07:55.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:55.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:07:55.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.210+0000 7f17277fe640 1 -- 172.21.15.138:0/3678555772 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1728070580 con 0x7f17280a5b40 2024-09-11T05:07:55.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f17277fe640 1 -- 172.21.15.138:0/3678555772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17280a5b40 msgr2=0x7f17280a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:55.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f17277fe640 1 --2- 172.21.15.138:0/3678555772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17280a5b40 0x7f17280a5f20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f17100097c0 tx=0x7f171001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9c0000e718b4202 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 1 -- 172.21.15.138:0/3678555772 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17100b0070 con 0x7f17280a5b40 2024-09-11T05:07:55.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:55.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: dump: 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:55.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 5 monclient: authenticate success, global_id 4294 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 1 -- 172.21.15.138:0/3678555772 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17100ab600 con 0x7f17280a5b40 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 4 set_mon_vals no callback set 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 1 -- 172.21.15.138:0/3678555772 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17100ab9d0 con 0x7f17280a5b40 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f1726ffd640 10 monclient: dump: 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.219 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.220 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.220 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.220 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.220 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.220 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 10 monclient: shutdown 2024-09-11T05:07:55.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:55.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 -- 172.21.15.138:0/3678555772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17280a5b40 msgr2=0x7f17280a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 --2- 172.21.15.138:0/3678555772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17280a5b40 0x7f17280a5f20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f17100097c0 tx=0x7f171001c360 comp rx=0 tx=0).stop 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 -- 172.21.15.138:0/3678555772 shutdown_connections 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 --2- 172.21.15.138:0/3678555772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17280a5b40 0x7f17280a5f20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 -- 172.21.15.138:0/3678555772 >> 172.21.15.138:0/3678555772 conn(0x7f17280aba10 msgr2=0x7f17280a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 -- 172.21.15.138:0/3678555772 shutdown_connections 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 -- 172.21.15.138:0/3678555772 wait complete. 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 10 monclient: build_initial_monmap 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 10 monclient: monmap: 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:55.225 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 librados: starting msgr at 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 librados: starting objecter 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 Processor -- start 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 -- start start 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 librados: setting wanted keys 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 1 librados: calling monclient init 2024-09-11T05:07:55.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.214+0000 7f172dfa7640 10 monclient: init 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f172809d840 0x7f1728140b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient(hunting): picked mon.noname-b con 0x7f172809d840 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17280a5b40 0x7f17281d1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient(hunting): picked mon.noname-c con 0x7f17280a5b40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17280a7130 0x7f172813e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient(hunting): picked mon.noname-a con 0x7f17280a7130 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1728144920 con 0x7f17280a7130 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17281447a0 con 0x7f172809d840 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1728144aa0 con 0x7f17280a5b40 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient(hunting): authenticate will time out at 958.656253s 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17280a7130 0x7f172813e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17280a5b40 0x7f17281d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17280a7130 0x7f172813e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46538/0 (socket says 172.21.15.138:46538) 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17280a5b40 0x7f17281d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58946/0 (socket says 172.21.15.138:58946) 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 1 -- 172.21.15.138:0/3776148067 learned_addr learned my addr 172.21.15.138:0/3776148067 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 10 monclient(hunting): get_auth_request con 0x7f17280a7130 auth_method 0 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient(hunting): get_auth_request con 0x7f17280a5b40 auth_method 0 2024-09-11T05:07:55.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f17277fe640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f172809d840 0x7f1728140b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f17277fe640 10 monclient(hunting): get_auth_request con 0x7f172809d840 auth_method 0 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f17277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f17277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f17277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 -- 172.21.15.138:0/3776148067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f172809d840 msgr2=0x7f1728140b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f172809d840 0x7f1728140b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 -- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17280a7130 msgr2=0x7f172813e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17280a7130 0x7f172813e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 monclient: found mon.noname-c 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 -- 172.21.15.138:0/3776148067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1728121fd0 con 0x7f17280a5b40 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1727fff640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17280a7130 0x7f172813e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 -- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17280a5b40 msgr2=0x7f17281d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f171ffff640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17280a5b40 0x7f17281d1270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f171800c410 tx=0x7f171800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 client.?.objecter ms_handle_connect 0x7f17280a5b40 2024-09-11T05:07:55.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 1 -- 172.21.15.138:0/3776148067 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17180119a0 con 0x7f17280a5b40 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: dump: 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.231 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: _finish_auth 0 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 1 -- 172.21.15.138:0/3776148067 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1718017070 con 0x7f17280a5b40 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 5 monclient: authenticate success, global_id 4313 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient: _renew_subs 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17281219a0 con 0x7f17280a5b40 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 1 -- 172.21.15.138:0/3776148067 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1718011d80 con 0x7f17280a5b40 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f1724ff9640 10 monclient: dump: 2024-09-11T05:07:55.232 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f17267fc640 4 set_mon_vals no callback set 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient: _renew_subs 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17281c9b30 con 0x7f17280a5b40 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 librados: init done 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.218+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17280a5f20 con 0x7f17280a5b40 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f1724ff9640 1 -- 172.21.15.138:0/3776148067 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f171801c460 con 0x7f17280a5b40 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f1724ff9640 1 --2- 172.21.15.138:0/3776148067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f170803cea0 0x7f170803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f1724ff9640 1 -- 172.21.15.138:0/3776148067 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f171804f0d0 con 0x7f17280a5b40 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f1724ff9640 10 client.4313.objecter ms_dispatch 0x7f1728000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f1724ff9640 3 client.4313.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f1724ff9640 3 client.4313.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f1724ff9640 20 client.4313.objecter dump_active .. 0 homeless 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f17277fe640 1 --2- 172.21.15.138:0/3776148067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f170803cea0 0x7f170803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f17277fe640 10 monclient: get_auth_request con 0x7f170803cea0 auth_method 0 2024-09-11T05:07:55.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f17277fe640 1 --2- 172.21.15.138:0/3776148067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f170803cea0 0x7f170803f360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f17100ab000 tx=0x7f1710008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.222+0000 7f1724ff9640 10 client.4313.objecter ms_handle_connect 0x7f170803cea0 2024-09-11T05:07:55.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.226+0000 7f1724ff9640 1 -- 172.21.15.138:0/3776148067 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f171801ac60 con 0x7f17280a5b40 2024-09-11T05:07:55.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.226+0000 7f1724ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.226+0000 7f1724ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:55.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.382+0000 7f172dfa7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:55.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.382+0000 7f172dfa7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:55.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.382+0000 7f172dfa7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.382+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1728069fa0 con 0x7f17280a5b40 2024-09-11T05:07:55.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.382+0000 7f1724ff9640 1 -- 172.21.15.138:0/3776148067 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10945 (secure 0 0 0) 0x7f171801c960 con 0x7f17280a5b40 2024-09-11T05:07:55.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.382+0000 7f1724ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:07:55.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.382+0000 7f1724ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:55.387 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:55.387 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":19,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","created":"2024-09-11T05:07:27.386635+0000","modified":"2024-09-11T05:07:50.526058+0000","last_up_change":"2024-09-11T05:07:40.695452+0000","last_in_change":"2024-09-11T05:07:31.585387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:07:40.718263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:07:46.826732+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79670fd2-b5aa-408e-8aa4-18a8fb848972","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":16171034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":16171034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":16171034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":16171034}]},"public_addr":"172.21.15.138:6800/16171034","cluster_addr":"172.21.15.138:6801/16171034","heartbeat_back_addr":"172.21.15.138:6803/16171034","heartbeat_front_addr":"172.21.15.138:6802/16171034","state":["exists","up"]},{"osd":1,"uuid":"e96fe3a6-3058-4c40-aa87-26a89572cdfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":4002974512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4002974512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":4002974512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":4002974512}]},"public_addr":"172.21.15.138:6808/4002974512","cluster_addr":"172.21.15.138:6809/4002974512","heartbeat_back_addr":"172.21.15.138:6811/4002974512","heartbeat_front_addr":"172.21.15.138:6810/4002974512","state":["exists","up"]},{"osd":2,"uuid":"a87dbce5-b78c-4070-90cc-d323fc444f4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":81047329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":81047329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":81047329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":81047329}]},"public_addr":"172.21.15.138:6804/81047329","cluster_addr":"172.21.15.138:6805/81047329","heartbeat_back_addr":"172.21.15.138:6807/81047329","heartbeat_front_addr":"172.21.15.138:6806/81047329","state":["exists","up"]},{"osd":3,"uuid":"e1a7ea3f-62f5-4d1f-b671-be183143eb20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3046884208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3046884208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3046884208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3046884208}]},"public_addr":"172.21.15.190:6809/3046884208","cluster_addr":"172.21.15.190:6810/3046884208","heartbeat_back_addr":"172.21.15.190:6812/3046884208","heartbeat_front_addr":"172.21.15.190:6811/3046884208","state":["exists","up"]},{"osd":4,"uuid":"a1aca44b-3dce-4f17-bc06-69bdca9cba04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":1118550024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1118550024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1118550024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":1118550024}]},"public_addr":"172.21.15.190:6801/1118550024","cluster_addr":"172.21.15.190:6802/1118550024","heartbeat_back_addr":"172.21.15.190:6804/1118550024","heartbeat_front_addr":"172.21.15.190:6803/1118550024","state":["exists","up"]},{"osd":5,"uuid":"87ad0149-d931-4ac6-8787-94f9a9609d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":189369151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":189369151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":189369151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":189369151}]},"public_addr":"172.21.15.190:6805/189369151","cluster_addr":"172.21.15.190:6806/189369151","heartbeat_back_addr":"172.21.15.190:6808/189369151","heartbeat_front_addr":"172.21.15.190:6807/189369151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.795496+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.359385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.802420+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.791133+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.377964+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.428799+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 10 librados: watch_flush enter 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 10 librados: watch_flush exit 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 20 client.4313.objecter shutdown clearing up homeless session... 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 10 client.4313.objecter successfully canceled tick 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f170803cea0 msgr2=0x7f170803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 --2- 172.21.15.138:0/3776148067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f170803cea0 0x7f170803f360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f17100ab000 tx=0x7f1710008040 comp rx=0 tx=0).stop 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 10 monclient: shutdown 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:55.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17280a5b40 msgr2=0x7f17281d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17280a5b40 0x7f17281d1270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f171800c410 tx=0x7f171800c8d0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 shutdown_connections 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 --2- 172.21.15.138:0/3776148067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f170803cea0 0x7f170803f360 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f17280a7130 0x7f172813e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f17280a5b40 0x7f17281d1270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 --2- 172.21.15.138:0/3776148067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f172809d840 0x7f1728140b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 >> 172.21.15.138:0/3776148067 conn(0x7f17280aba10 msgr2=0x7f17280a92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 shutdown_connections 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 -- 172.21.15.138:0/3776148067 wait complete. 2024-09-11T05:07:55.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.386+0000 7f172dfa7640 1 librados: shutdown 2024-09-11T05:07:55.404 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T05:07:40.718263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T05:07:46.826732+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-11T05:07:55.404 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-11T05:07:55.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.510+0000 7f31fa703640 10 monclient: get_monmap_and_config 2024-09-11T05:07:55.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.510+0000 7f31fa703640 10 monclient: build_initial_monmap 2024-09-11T05:07:55.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.510+0000 7f31fa703640 10 monclient: monmap: 2024-09-11T05:07:55.515 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:55.515 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.515 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:55.516156+0000 2024-09-11T05:07:55.516 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:55.516156+0000 2024-09-11T05:07:55.516 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:55.516 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.516 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:55.516 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:55.516 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:55.517 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:55.517 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:55.517 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.510+0000 7f31fa703640 1 Processor -- start 2024-09-11T05:07:55.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.510+0000 7f31fa703640 1 -- start start 2024-09-11T05:07:55.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.510+0000 7f31fa703640 10 monclient: init 2024-09-11T05:07:55.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:55.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T05:07:55.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f31f409da20 0x7f31f40a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient(hunting): picked mon.noname-c con 0x7f31f409da20 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:55.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f31f40a1f40 0x7f31f40a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient(hunting): picked mon.noname-a con 0x7f31f40a1f40 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:55.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31f4136170 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:55.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f4074a80 con 0x7f31f4136170 2024-09-11T05:07:55.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f31f40a2db0 con 0x7f31f40a1f40 2024-09-11T05:07:55.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f31f40a3070 con 0x7f31f409da20 2024-09-11T05:07:55.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:55.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31fa703640 10 monclient(hunting): authenticate will time out at 958.952265s 2024-09-11T05:07:55.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f31f40a1f40 0x7f31f40a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f31f40a1f40 0x7f31f40a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46552/0 (socket says 172.21.15.138:46552) 2024-09-11T05:07:55.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 1 -- 172.21.15.138:0/3982639824 learned_addr learned my addr 172.21.15.138:0/3982639824 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:55.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient(hunting): get_auth_request con 0x7f31f40a1f40 auth_method 0 2024-09-11T05:07:55.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f3fff640 1 --2- 172.21.15.138:0/3982639824 >> v2:172.21.15.190:3300/0 conn(0x7f31f409da20 0x7f31f40a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2024-09-11T05:07:55.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 1 -- 172.21.15.138:0/3982639824 >> v2:172.21.15.190:3300/0 conn(0x7f31f409da20 msgr2=0x7f31f40a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:55.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 1 --2- 172.21.15.138:0/3982639824 >> v2:172.21.15.190:3300/0 conn(0x7f31f409da20 0x7f31f40a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 1 -- 172.21.15.138:0/3982639824 >> v1:172.21.15.190:6789/0 conn(0x7f31f4136170 legacy=0x7f31f41365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:55.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:55.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 1 monclient: found mon.noname-a 2024-09-11T05:07:55.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:55.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:55.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f37fe640 1 -- 172.21.15.138:0/3982639824 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f40a3350 con 0x7f31f40a1f40 2024-09-11T05:07:55.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f3fff640 10 monclient: get_auth_request con 0x7f31f409da20 auth_method 0 2024-09-11T05:07:55.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.514+0000 7f31f3fff640 1 --2- 172.21.15.138:0/3982639824 >> v2:172.21.15.190:3300/0 conn(0x7f31f409da20 0x7f31f40a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:55.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 -- 172.21.15.138:0/3982639824 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 msgr2=0x7f31f40a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:55.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 --2- 172.21.15.138:0/3982639824 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f40a23d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f31f4069440 tx=0x7f31e801c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11201ddae2900586 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 1 -- 172.21.15.138:0/3982639824 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e80b0070 con 0x7f31f40a1f40 2024-09-11T05:07:55.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:55.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: dump: 2024-09-11T05:07:55.539 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.539 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.539 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.539 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.539 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.539 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.540 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.540 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.540 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.540 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:55.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:55.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:55.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 5 monclient: authenticate success, global_id 4320 2024-09-11T05:07:55.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 1 -- 172.21.15.138:0/3982639824 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e80ab600 con 0x7f31f40a1f40 2024-09-11T05:07:55.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:55.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 1 -- 172.21.15.138:0/3982639824 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e80ab9d0 con 0x7f31f40a1f40 2024-09-11T05:07:55.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:55.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 4 set_mon_vals no callback set 2024-09-11T05:07:55.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:55.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f2ffd640 10 monclient: dump: 2024-09-11T05:07:55.542 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.542 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.542 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.543 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.543 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.543 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.543 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.543 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.543 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.543 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient: shutdown 2024-09-11T05:07:55.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:55.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- 172.21.15.138:0/3982639824 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 msgr2=0x7f31f40a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 --2- 172.21.15.138:0/3982639824 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f40a23d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f31f4069440 tx=0x7f31e801c1c0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- 172.21.15.138:0/3982639824 shutdown_connections 2024-09-11T05:07:55.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 --2- 172.21.15.138:0/3982639824 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f40a23d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 --2- 172.21.15.138:0/3982639824 >> v2:172.21.15.190:3300/0 conn(0x7f31f409da20 0x7f31f40a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- 172.21.15.138:0/3982639824 >> 172.21.15.138:0/3982639824 conn(0x7f31f40a3b00 msgr2=0x7f31f40a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:55.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- 172.21.15.138:0/3982639824 shutdown_connections 2024-09-11T05:07:55.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- 172.21.15.138:0/3982639824 wait complete. 2024-09-11T05:07:55.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient: build_initial_monmap 2024-09-11T05:07:55.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient: monmap: 2024-09-11T05:07:55.546 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:55.546 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:55.546 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:55.546 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:55.546 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:55.546 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.547 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:55.547 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:55.547 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:55.547 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 librados: starting msgr at 2024-09-11T05:07:55.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 librados: starting objecter 2024-09-11T05:07:55.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 Processor -- start 2024-09-11T05:07:55.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- start start 2024-09-11T05:07:55.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 librados: setting wanted keys 2024-09-11T05:07:55.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 librados: calling monclient init 2024-09-11T05:07:55.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient: init 2024-09-11T05:07:55.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:55.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:55.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f409da20 0x7f31f412fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient(hunting): picked mon.noname-c con 0x7f31f409da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:55.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f41371f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient(hunting): picked mon.noname-a con 0x7f31f40a1f40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:55.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f4136170 0x7f31f4137730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient(hunting): picked mon.noname-b con 0x7f31f4136170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:55.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f31f41410c0 con 0x7f31f40a1f40 2024-09-11T05:07:55.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f31f4140f40 con 0x7f31f4136170 2024-09-11T05:07:55.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f31f4141240 con 0x7f31f409da20 2024-09-11T05:07:55.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:55.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31fa703640 10 monclient(hunting): authenticate will time out at 958.958955s 2024-09-11T05:07:55.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f41371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f409da20 0x7f31f412fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f41371f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46564/0 (socket says 172.21.15.138:46564) 2024-09-11T05:07:55.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 -- 172.21.15.138:0/238484877 learned_addr learned my addr 172.21.15.138:0/238484877 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:55.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient(hunting): get_auth_request con 0x7f31f40a1f40 auth_method 0 2024-09-11T05:07:55.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f3fff640 10 monclient(hunting): get_auth_request con 0x7f31f409da20 auth_method 0 2024-09-11T05:07:55.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-09-11T05:07:55.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 -- 172.21.15.138:0/238484877 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f409da20 msgr2=0x7f31f412fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 --2- 172.21.15.138:0/238484877 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f409da20 0x7f31f412fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 -- 172.21.15.138:0/238484877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f4136170 msgr2=0x7f31f4137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 --2- 172.21.15.138:0/238484877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f4136170 0x7f31f4137730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:55.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 monclient: found mon.noname-a 2024-09-11T05:07:55.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:55.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:55.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.518+0000 7f31f37fe640 1 -- 172.21.15.138:0/238484877 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f4134200 con 0x7f31f40a1f40 2024-09-11T05:07:55.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f37fe640 1 -- 172.21.15.138:0/238484877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 msgr2=0x7f31f41371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:55.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f37fe640 1 --2- 172.21.15.138:0/238484877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f41371f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f31e801a870 tx=0x7f31e8018f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 client.?.objecter ms_handle_connect 0x7f31f40a1f40 2024-09-11T05:07:55.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:55.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 1 -- 172.21.15.138:0/238484877 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e80b0040 con 0x7f31f40a1f40 2024-09-11T05:07:55.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:55.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: dump: 2024-09-11T05:07:55.556 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.557 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.557 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.557 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.557 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.557 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.557 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.557 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.558 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.558 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: _finish_auth 0 2024-09-11T05:07:55.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:07:55.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:55.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 5 monclient: authenticate success, global_id 4323 2024-09-11T05:07:55.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 1 -- 172.21.15.138:0/238484877 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e80aa5b0 con 0x7f31f40a1f40 2024-09-11T05:07:55.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 10 monclient: _renew_subs 2024-09-11T05:07:55.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:55.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f413aae0 con 0x7f31f40a1f40 2024-09-11T05:07:55.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:55.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 1 -- 172.21.15.138:0/238484877 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e80aa9a0 con 0x7f31f40a1f40 2024-09-11T05:07:55.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:55.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 monclient: dump: 2024-09-11T05:07:55.560 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.560 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.560 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.560 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.560 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.561 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.561 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.561 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.561 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.561 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:55.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 10 monclient: _renew_subs 2024-09-11T05:07:55.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:55.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f4139790 con 0x7f31f40a1f40 2024-09-11T05:07:55.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 1 librados: init done 2024-09-11T05:07:55.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f27fc640 4 set_mon_vals no callback set 2024-09-11T05:07:55.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:55.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f40a1a00 con 0x7f31f40a1f40 2024-09-11T05:07:55.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 1 -- 172.21.15.138:0/238484877 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f31e80aab40 con 0x7f31f40a1f40 2024-09-11T05:07:55.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 1 --2- 172.21.15.138:0/238484877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31e003cea0 0x7f31e003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 1 -- 172.21.15.138:0/238484877 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f31e80e9050 con 0x7f31f40a1f40 2024-09-11T05:07:55.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 10 client.4323.objecter ms_dispatch 0x7f31f4000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:55.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 3 client.4323.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:55.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 3 client.4323.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:55.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f0ff9640 20 client.4323.objecter dump_active .. 0 homeless 2024-09-11T05:07:55.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f3fff640 1 --2- 172.21.15.138:0/238484877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31e003cea0 0x7f31e003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.522+0000 7f31f3fff640 10 monclient: get_auth_request con 0x7f31e003cea0 auth_method 0 2024-09-11T05:07:55.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.526+0000 7f31f0ff9640 10 client.4323.objecter ms_handle_connect 0x7f31e003cea0 2024-09-11T05:07:55.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.526+0000 7f31f3fff640 1 --2- 172.21.15.138:0/238484877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31e003cea0 0x7f31e003f360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00d620 tx=0x7f31ec007810 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.526+0000 7f31f0ff9640 1 -- 172.21.15.138:0/238484877 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f31f4074a80 con 0x7f31f40a1f40 2024-09-11T05:07:55.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.526+0000 7f31f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.526+0000 7f31f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:55.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.686+0000 7f31fa703640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T05:07:55.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.686+0000 7f31fa703640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T05:07:55.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.686+0000 7f31fa703640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:55.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.686+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f31f409e170 con 0x7f31f40a1f40 2024-09-11T05:07:55.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.686+0000 7f31f0ff9640 1 -- 172.21.15.138:0/238484877 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (secure 0 0 0) 0x7f31e80a93e0 con 0x7f31f40a1f40 2024-09-11T05:07:55.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.686+0000 7f31f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T05:07:55.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.686+0000 7f31f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:55.691 INFO:teuthology.orchestra.run.smithi138.stdout:pg_num: 1 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 10 librados: watch_flush enter 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 10 librados: watch_flush exit 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 20 client.4323.objecter shutdown clearing up homeless session... 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 10 client.4323.objecter successfully canceled tick 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31e003cea0 msgr2=0x7f31e003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 --2- 172.21.15.138:0/238484877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31e003cea0 0x7f31e003f360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00d620 tx=0x7f31ec007810 comp rx=0 tx=0).stop 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 10 monclient: shutdown 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:55.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 msgr2=0x7f31f41371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 --2- 172.21.15.138:0/238484877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f41371f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f31e801a870 tx=0x7f31e8018f60 comp rx=0 tx=0).stop 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 shutdown_connections 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 --2- 172.21.15.138:0/238484877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f31e003cea0 0x7f31e003f360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 --2- 172.21.15.138:0/238484877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31f4136170 0x7f31f4137730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 --2- 172.21.15.138:0/238484877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f31f40a1f40 0x7f31f41371f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 --2- 172.21.15.138:0/238484877 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f31f409da20 0x7f31f412fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 >> 172.21.15.138:0/238484877 conn(0x7f31f40a3b00 msgr2=0x7f31f4132dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 shutdown_connections 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 -- 172.21.15.138:0/238484877 wait complete. 2024-09-11T05:07:55.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.690+0000 7f31fa703640 1 librados: shutdown 2024-09-11T05:07:55.708 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.814+0000 7f2c13839640 10 monclient: get_monmap_and_config 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.814+0000 7f2c13839640 10 monclient: build_initial_monmap 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.814+0000 7f2c13839640 10 monclient: monmap: 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:55.820795+0000 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:55.820795+0000 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:55.819 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:55.820 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:55.820 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:55.820 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:55.820 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 1 Processor -- start 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 1 -- start start 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient: init 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2c0c1352f0 0x7f2c0c1356d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient(hunting): picked mon.noname-a con 0x7f2c0c1352f0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2c0c0a1b00 0x7f2c0c138380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient(hunting): picked mon.noname-c con 0x7f2c0c0a1b00 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c0c134e00 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c0c0a2740 con 0x7f2c0c134e00 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f2c0c0a2a40 con 0x7f2c0c1352f0 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2c0c0a2d00 con 0x7f2c0c0a1b00 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c13839640 10 monclient(hunting): authenticate will time out at 959.256906s 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2c0c1352f0 0x7f2c0c1356d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2c0c1352f0 0x7f2c0c1356d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46572/0 (socket says 172.21.15.138:46572) 2024-09-11T05:07:55.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 1 -- 172.21.15.138:0/2397802324 learned_addr learned my addr 172.21.15.138:0/2397802324 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient(hunting): get_auth_request con 0x7f2c0c1352f0 auth_method 0 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c11dd6640 1 --1- 172.21.15.138:0/2397802324 >> v1:172.21.15.138:6789/0 conn(0x7f2c0c134e00 0x7f2c0c1388c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38458/0 (socket says 172.21.15.138:38458) 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c10dd4640 1 --2- 172.21.15.138:0/2397802324 >> v2:172.21.15.190:3300/0 conn(0x7f2c0c0a1b00 0x7f2c0c138380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 1 -- 172.21.15.138:0/2397802324 >> v2:172.21.15.190:3300/0 conn(0x7f2c0c0a1b00 msgr2=0x7f2c0c138380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 1 --2- 172.21.15.138:0/2397802324 >> v2:172.21.15.190:3300/0 conn(0x7f2c0c0a1b00 0x7f2c0c138380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 1 -- 172.21.15.138:0/2397802324 >> v1:172.21.15.138:6789/0 conn(0x7f2c0c134e00 legacy=0x7f2c0c1388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:55.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient: _finish_hunting 0 2024-09-11T05:07:55.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 1 monclient: found mon.noname-a 2024-09-11T05:07:55.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:55.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:55.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.818+0000 7f2c115d5640 1 -- 172.21.15.138:0/2397802324 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c0a36f0 con 0x7f2c0c1352f0 2024-09-11T05:07:55.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c115d5640 1 -- 172.21.15.138:0/2397802324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 msgr2=0x7f2c0c1356d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:55.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c115d5640 1 --2- 172.21.15.138:0/2397802324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c1356d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c06e390 tx=0x7f2bf401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230aba8a9e8f72c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 1 -- 172.21.15.138:0/2397802324 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c09de90 con 0x7f2c0c1352f0 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: dump: 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: _finish_auth 0 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 5 monclient: authenticate success, global_id 4326 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 1 -- 172.21.15.138:0/2397802324 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf40a6500 con 0x7f2c0c1352f0 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 4 set_mon_vals no callback set 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 1 -- 172.21.15.138:0/2397802324 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf40a68d0 con 0x7f2c0c1352f0 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c0bfff640 10 monclient: dump: 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.830 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient: shutdown 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- 172.21.15.138:0/2397802324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 msgr2=0x7f2c0c1356d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 --2- 172.21.15.138:0/2397802324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c1356d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c06e390 tx=0x7f2bf401c760 comp rx=0 tx=0).stop 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- 172.21.15.138:0/2397802324 shutdown_connections 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 --2- 172.21.15.138:0/2397802324 >> v2:172.21.15.190:3300/0 conn(0x7f2c0c0a1b00 0x7f2c0c138380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 --2- 172.21.15.138:0/2397802324 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c1356d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- 172.21.15.138:0/2397802324 >> 172.21.15.138:0/2397802324 conn(0x7f2c0c0a3b00 msgr2=0x7f2c0c0a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- 172.21.15.138:0/2397802324 shutdown_connections 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- 172.21.15.138:0/2397802324 wait complete. 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient: build_initial_monmap 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient: monmap: 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:55.832 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 librados: starting msgr at 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 librados: starting objecter 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 Processor -- start 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- start start 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 librados: setting wanted keys 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 librados: calling monclient init 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient: init 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c0c134e00 0x7f2c0c12eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient(hunting): picked mon.noname-b con 0x7f2c0c134e00 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c12eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient(hunting): picked mon.noname-a con 0x7f2c0c1352f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2c0c1394b0 0x7f2c0c139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient(hunting): picked mon.noname-c con 0x7f2c0c1394b0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c142f40 con 0x7f2c0c1352f0 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c1391e0 con 0x7f2c0c134e00 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0c1430c0 con 0x7f2c0c1394b0 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.822+0000 7f2c13839640 10 monclient(hunting): authenticate will time out at 959.263899s 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c12eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2c0c1394b0 0x7f2c0c139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c12eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46582/0 (socket says 172.21.15.138:46582) 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 1 -- 172.21.15.138:0/3629863121 learned_addr learned my addr 172.21.15.138:0/3629863121 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:55.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 10 monclient(hunting): get_auth_request con 0x7f2c0c1352f0 auth_method 0 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient(hunting): get_auth_request con 0x7f2c0c1394b0 auth_method 0 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:55.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c115d5640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c0c134e00 0x7f2c0c12eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 -- 172.21.15.138:0/3629863121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c0c134e00 msgr2=0x7f2c0c12eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c0c134e00 0x7f2c0c12eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 -- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 msgr2=0x7f2c0c12eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c12eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient: _finish_hunting 0 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 monclient: found mon.noname-c 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 -- 172.21.15.138:0/3629863121 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c0a6660 con 0x7f2c0c1394b0 2024-09-11T05:07:55.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c10dd4640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c12eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:55.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 -- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2c0c1394b0 msgr2=0x7f2c0c139940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c11dd6640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2c0c1394b0 0x7f2c0c139940 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400c410 tx=0x7f2c0400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 client.?.objecter ms_handle_connect 0x7f2c0c1394b0 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 1 -- 172.21.15.138:0/3629863121 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04017070 con 0x7f2c0c1394b0 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: dump: 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.839 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 1 -- 172.21.15.138:0/3629863121 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c040076f0 con 0x7f2c0c1394b0 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 1 -- 172.21.15.138:0/3629863121 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c040059f0 con 0x7f2c0c1394b0 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 5 monclient: authenticate success, global_id 4319 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 10 monclient: _renew_subs 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c0c09e1f0 con 0x7f2c0c1394b0 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c0b7fe640 4 set_mon_vals no callback set 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 10 monclient: dump: 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:55.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 10 monclient: _renew_subs 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c0c13bdb0 con 0x7f2c0c1394b0 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 1 librados: init done 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c0c1356d0 con 0x7f2c0c1394b0 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 1 -- 172.21.15.138:0/3629863121 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f2c04005c10 con 0x7f2c0c1394b0 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.826+0000 7f2c09ffb640 1 --2- 172.21.15.138:0/3629863121 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2bec03cf20 0x7f2bec03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c09ffb640 1 -- 172.21.15.138:0/3629863121 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f2c0404e8d0 con 0x7f2c0c1394b0 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c09ffb640 10 client.4319.objecter ms_dispatch 0x7f2c0c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c09ffb640 3 client.4319.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:55.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c09ffb640 3 client.4319.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:55.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c09ffb640 20 client.4319.objecter dump_active .. 0 homeless 2024-09-11T05:07:55.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c115d5640 1 --2- 172.21.15.138:0/3629863121 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2bec03cf20 0x7f2bec03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:55.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c115d5640 10 monclient: get_auth_request con 0x7f2bec03cf20 auth_method 0 2024-09-11T05:07:55.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c115d5640 1 --2- 172.21.15.138:0/3629863121 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2bec03cf20 0x7f2bec03f3e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c074bf0 tx=0x7f2bf40b1040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:55.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.830+0000 7f2c09ffb640 10 client.4319.objecter ms_handle_connect 0x7f2bec03cf20 2024-09-11T05:07:55.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.834+0000 7f2c09ffb640 1 -- 172.21.15.138:0/3629863121 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f2c0c09e1f0 con 0x7f2c0c1394b0 2024-09-11T05:07:55.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.834+0000 7f2c09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:55.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.834+0000 7f2c09ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:55.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T05:07:55.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T05:07:55.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:55.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f2c0c0a6b20 con 0x7f2c0c1394b0 2024-09-11T05:07:55.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c09ffb640 1 -- 172.21.15.138:0/3629863121 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) ==== 92+0+10 (secure 0 0 0) 0x7f2c0c0a6660 con 0x7f2c0c1394b0 2024-09-11T05:07:55.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T05:07:55.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c09ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:55.993 INFO:teuthology.orchestra.run.smithi138.stdout:pg_num: 8 2024-09-11T05:07:55.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 10 librados: watch_flush enter 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 10 librados: watch_flush exit 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 20 client.4319.objecter shutdown clearing up homeless session... 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 10 client.4319.objecter successfully canceled tick 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2bec03cf20 msgr2=0x7f2bec03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 --2- 172.21.15.138:0/3629863121 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2bec03cf20 0x7f2bec03f3e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c074bf0 tx=0x7f2bf40b1040 comp rx=0 tx=0).stop 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 10 monclient: shutdown 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2c0c1394b0 msgr2=0x7f2c0c139940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2c0c1394b0 0x7f2c0c139940 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400c410 tx=0x7f2c0400c8d0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 shutdown_connections 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 --2- 172.21.15.138:0/3629863121 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2bec03cf20 0x7f2bec03f3e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2c0c1394b0 0x7f2c0c139940 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2c0c1352f0 0x7f2c0c12eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 --2- 172.21.15.138:0/3629863121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c0c134e00 0x7f2c0c12eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:55.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 >> 172.21.15.138:0/3629863121 conn(0x7f2c0c0a3b00 msgr2=0x7f2c0c0a5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:55.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 shutdown_connections 2024-09-11T05:07:55.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 -- 172.21.15.138:0/3629863121 wait complete. 2024-09-11T05:07:55.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:55.990+0000 7f2c13839640 1 librados: shutdown 2024-09-11T05:07:56.010 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-11T05:07:56.011 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:56.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 10 monclient: get_monmap_and_config 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 10 monclient: build_initial_monmap 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 10 monclient: monmap: 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:56.119503+0000 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:56.119503+0000 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:56.118 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 1 Processor -- start 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 1 -- start start 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 10 monclient: init 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97c40a5b40 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f97c40a7130 0x7f97c413db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.114+0000 7f97ca7da640 10 monclient(hunting): picked mon.noname-a con 0x7f97c40a7130 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c4140090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 10 monclient(hunting): picked mon.noname-b con 0x7f97c409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c409d490 con 0x7f97c40a5b40 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f97c409e110 con 0x7f97c40a7130 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f97c4070580 con 0x7f97c409d840 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97ca7da640 10 monclient(hunting): authenticate will time out at 959.556089s 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f97c40a7130 0x7f97c413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c8d77640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c4140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c3fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f97c40a5b40 0x7f97c40a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38462/0 (socket says 172.21.15.138:38462) 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f97c40a7130 0x7f97c413db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46586/0 (socket says 172.21.15.138:46586) 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c3fff640 1 -- 172.21.15.138:0/1508597214 learned_addr learned my addr 172.21.15.138:0/1508597214 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:56.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient(hunting): get_auth_request con 0x7f97c40a7130 auth_method 0 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c8d77640 10 monclient(hunting): get_auth_request con 0x7f97c409d840 auth_method 0 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c8d77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c8d77640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c8d77640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c8d77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c8d77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97c8d77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2024-09-11T05:07:56.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 1 -- 172.21.15.138:0/1508597214 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 msgr2=0x7f97c4140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:56.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 1 --2- 172.21.15.138:0/1508597214 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c4140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 1 -- 172.21.15.138:0/1508597214 >> v1:172.21.15.138:6789/0 conn(0x7f97c40a5b40 legacy=0x7f97c40a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:56.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:56.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 1 monclient: found mon.noname-a 2024-09-11T05:07:56.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:56.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.118+0000 7f97bb7fe640 1 -- 172.21.15.138:0/1508597214 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c409cb80 con 0x7f97c40a7130 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97bb7fe640 1 -- 172.21.15.138:0/1508597214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 msgr2=0x7f97c413db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97bb7fe640 1 --2- 172.21.15.138:0/1508597214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 0x7f97c413db40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0097c0 tx=0x7f97ac01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec1ecb7c03106cd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 1 -- 172.21.15.138:0/1508597214 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0b0070 con 0x7f97c40a7130 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: dump: 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.128 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 5 monclient: authenticate success, global_id 4332 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 1 -- 172.21.15.138:0/1508597214 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac0ab600 con 0x7f97c40a7130 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 4 set_mon_vals no callback set 2024-09-11T05:07:56.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 1 -- 172.21.15.138:0/1508597214 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0ab9d0 con 0x7f97c40a7130 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c37fe640 10 monclient: dump: 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient: shutdown 2024-09-11T05:07:56.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- 172.21.15.138:0/1508597214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 msgr2=0x7f97c413db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1508597214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 0x7f97c413db40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0097c0 tx=0x7f97ac01c360 comp rx=0 tx=0).stop 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- 172.21.15.138:0/1508597214 shutdown_connections 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1508597214 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c4140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1508597214 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 0x7f97c413db40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- 172.21.15.138:0/1508597214 >> 172.21.15.138:0/1508597214 conn(0x7f97c40aba10 msgr2=0x7f97c40a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- 172.21.15.138:0/1508597214 shutdown_connections 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- 172.21.15.138:0/1508597214 wait complete. 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient: build_initial_monmap 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient: monmap: 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:56.131 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 librados: starting msgr at 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 librados: starting objecter 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 Processor -- start 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- start start 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 librados: setting wanted keys 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 librados: calling monclient init 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient: init 2024-09-11T05:07:56.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:56.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:07:56.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c413dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient(hunting): picked mon.noname-c con 0x7f97c409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:56.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c40a5b40 0x7f97c4121b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient(hunting): picked mon.noname-b con 0x7f97c40a5b40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:56.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 0x7f97c41389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient(hunting): picked mon.noname-a con 0x7f97c40a7130 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:56.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4143180 con 0x7f97c40a7130 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4143000 con 0x7f97c40a5b40 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f97c4143300 con 0x7f97c409d840 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97ca7da640 10 monclient(hunting): authenticate will time out at 959.562964s 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c413dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c413dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58968/0 (socket says 172.21.15.138:58968) 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 -- 172.21.15.138:0/1412533656 learned_addr learned my addr 172.21.15.138:0/1412533656 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient(hunting): get_auth_request con 0x7f97c409d840 auth_method 0 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 0x7f97c41389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 10 monclient(hunting): get_auth_request con 0x7f97c40a7130 auth_method 0 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 -- 172.21.15.138:0/1412533656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c40a5b40 msgr2=0x7f97c4121b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c40a5b40 0x7f97c4121b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 -- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 msgr2=0x7f97c41389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 0x7f97c41389d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:56.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 monclient: found mon.noname-c 2024-09-11T05:07:56.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:56.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:56.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c3fff640 1 -- 172.21.15.138:0/1412533656 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c40a2330 con 0x7f97c409d840 2024-09-11T05:07:56.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.122+0000 7f97c8d77640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 0x7f97c41389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:56.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c3fff640 1 -- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 msgr2=0x7f97c413dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:56.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c3fff640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c413dea0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c4a0 tx=0x7f97b400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 client.?.objecter ms_handle_connect 0x7f97c409d840 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 1 -- 172.21.15.138:0/1412533656 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4017070 con 0x7f97c409d840 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: dump: 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.142 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: _finish_auth 0 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: _check_auth_tickets 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 1 -- 172.21.15.138:0/1412533656 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b4007070 con 0x7f97c409d840 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 5 monclient: authenticate success, global_id 4325 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 10 monclient: _renew_subs 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 1 -- 172.21.15.138:0/1412533656 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4007460 con 0x7f97c409d840 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c413f430 con 0x7f97c409d840 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c2ffd640 4 set_mon_vals no callback set 2024-09-11T05:07:56.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 monclient: dump: 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 10 monclient: _renew_subs 2024-09-11T05:07:56.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c4122ac0 con 0x7f97c409d840 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 1 librados: init done 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c40a5f20 con 0x7f97c409d840 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 1 -- 172.21.15.138:0/1412533656 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f97b401b3b0 con 0x7f97c409d840 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 1 --2- 172.21.15.138:0/1412533656 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97a403cef0 0x7f97a403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 1 -- 172.21.15.138:0/1412533656 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (secure 0 0 0) 0x7f97b404eae0 con 0x7f97c409d840 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 10 client.4325.objecter ms_dispatch 0x7f97c4000ba0 osd_map(19..19 src has 1..19) 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 3 client.4325.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 3 client.4325.objecter handle_osd_map decoding full epoch 19 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97c17fa640 20 client.4325.objecter dump_active .. 0 homeless 2024-09-11T05:07:56.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97bb7fe640 1 --2- 172.21.15.138:0/1412533656 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97a403cef0 0x7f97a403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.126+0000 7f97bb7fe640 10 monclient: get_auth_request con 0x7f97a403cef0 auth_method 0 2024-09-11T05:07:56.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.130+0000 7f97bb7fe640 1 --2- 172.21.15.138:0/1412533656 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97a403cef0 0x7f97a403f3b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97c40524c0 tx=0x7f97ac004740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.130+0000 7f97c17fa640 10 client.4325.objecter ms_handle_connect 0x7f97a403cef0 2024-09-11T05:07:56.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.130+0000 7f97c17fa640 1 -- 172.21.15.138:0/1412533656 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f97b401be80 con 0x7f97c409d840 2024-09-11T05:07:56.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.130+0000 7f97c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.130+0000 7f97c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.286+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97c4122f40 con 0x7f97a403cef0 2024-09-11T05:07:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97c17fa640 1 -- 172.21.15.138:0/1412533656 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34153 (secure 0 0 0) 0x7f97c4122f40 con 0x7f97a403cef0 2024-09-11T05:07:56.293 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:07:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 10 librados: watch_flush enter 2024-09-11T05:07:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 10 librados: watch_flush exit 2024-09-11T05:07:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 20 client.4325.objecter shutdown clearing up homeless session... 2024-09-11T05:07:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 10 client.4325.objecter successfully canceled tick 2024-09-11T05:07:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97a403cef0 msgr2=0x7f97a403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1412533656 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97a403cef0 0x7f97a403f3b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97c40524c0 tx=0x7f97ac004740 comp rx=0 tx=0).stop 2024-09-11T05:07:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 10 monclient: shutdown 2024-09-11T05:07:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 msgr2=0x7f97c413dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c413dea0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c4a0 tx=0x7f97b400c960 comp rx=0 tx=0).stop 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 shutdown_connections 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1412533656 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97a403cef0 0x7f97a403f3b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97c40a7130 0x7f97c41389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97c40a5b40 0x7f97c4121b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 --2- 172.21.15.138:0/1412533656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97c409d840 0x7f97c413dea0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 >> 172.21.15.138:0/1412533656 conn(0x7f97c40aba10 msgr2=0x7f97c40a9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 shutdown_connections 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 -- 172.21.15.138:0/1412533656 wait complete. 2024-09-11T05:07:56.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.290+0000 7f97ca7da640 1 librados: shutdown 2024-09-11T05:07:56.310 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-11T05:07:56.242895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163296,"kb_used_data":1088,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067744,"statfs":{"total":579820584960,"available":579653369856,"internally_reserved":0,"allocated":1114112,"data_stored":1375018,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.578865"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.536048+0000","last_change":"2024-09-11T05:07:50.536048+0000","last_active":"2024-09-11T05:07:50.536048+0000","last_peered":"2024-09-11T05:07:50.536048+0000","last_clean":"2024-09-11T05:07:50.536048+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:07:50.536048+0000","last_undegraded":"2024-09-11T05:07:50.536048+0000","last_fullsized":"2024-09-11T05:07:50.536048+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:33.527347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543840+0000","last_change":"2024-09-11T05:07:50.543840+0000","last_active":"2024-09-11T05:07:50.543840+0000","last_peered":"2024-09-11T05:07:50.543840+0000","last_clean":"2024-09-11T05:07:50.543840+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:07:50.543840+0000","last_undegraded":"2024-09-11T05:07:50.543840+0000","last_fullsized":"2024-09-11T05:07:50.543840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:09.809820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.535722+0000","last_change":"2024-09-11T05:07:50.535722+0000","last_active":"2024-09-11T05:07:50.535722+0000","last_peered":"2024-09-11T05:07:50.535722+0000","last_clean":"2024-09-11T05:07:50.535722+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:07:50.535722+0000","last_undegraded":"2024-09-11T05:07:50.535722+0000","last_fullsized":"2024-09-11T05:07:50.535722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:07.310720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543876+0000","last_change":"2024-09-11T05:07:50.543876+0000","last_active":"2024-09-11T05:07:50.543876+0000","last_peered":"2024-09-11T05:07:50.543876+0000","last_clean":"2024-09-11T05:07:50.543876+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:07:50.543876+0000","last_undegraded":"2024-09-11T05:07:50.543876+0000","last_fullsized":"2024-09-11T05:07:50.543876+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:47:12.140333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543322+0000","last_change":"2024-09-11T05:07:50.541650+0000","last_active":"2024-09-11T05:07:50.543322+0000","last_peered":"2024-09-11T05:07:50.543322+0000","last_clean":"2024-09-11T05:07:50.543322+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:07:50.543322+0000","last_undegraded":"2024-09-11T05:07:50.543322+0000","last_fullsized":"2024-09-11T05:07:50.543322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:47.451082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.533338+0000","last_change":"2024-09-11T05:07:50.533338+0000","last_active":"2024-09-11T05:07:50.533338+0000","last_peered":"2024-09-11T05:07:50.533338+0000","last_clean":"2024-09-11T05:07:50.533338+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:07:50.533338+0000","last_undegraded":"2024-09-11T05:07:50.533338+0000","last_fullsized":"2024-09-11T05:07:50.533338+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:04.573351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.535448+0000","last_change":"2024-09-11T05:07:50.535448+0000","last_active":"2024-09-11T05:07:50.535448+0000","last_peered":"2024-09-11T05:07:50.535448+0000","last_clean":"2024-09-11T05:07:50.535448+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:07:50.535448+0000","last_undegraded":"2024-09-11T05:07:50.535448+0000","last_fullsized":"2024-09-11T05:07:50.535448+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:57.570691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.540810+0000","last_change":"2024-09-11T05:07:50.540810+0000","last_active":"2024-09-11T05:07:50.540810+0000","last_peered":"2024-09-11T05:07:50.540810+0000","last_clean":"2024-09-11T05:07:50.540810+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:07:50.540810+0000","last_undegraded":"2024-09-11T05:07:50.540810+0000","last_fullsized":"2024-09-11T05:07:50.540810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:39.133232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.534426+0000","last_change":"2024-09-11T05:07:43.495116+0000","last_active":"2024-09-11T05:07:50.534426+0000","last_peered":"2024-09-11T05:07:50.534426+0000","last_clean":"2024-09-11T05:07:50.534426+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:07:50.534426+0000","last_undegraded":"2024-09-11T05:07:50.534426+0000","last_fullsized":"2024-09-11T05:07:50.534426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_clean_scrub_stamp":"2024-09-11T05:07:41.703163+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:01.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":581782,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":581782,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:56.311 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:07:56.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.422+0000 7f99df638640 10 monclient: get_monmap_and_config 2024-09-11T05:07:56.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.422+0000 7f99df638640 10 monclient: build_initial_monmap 2024-09-11T05:07:56.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.422+0000 7f99df638640 10 monclient: monmap: 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:56.429351+0000 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:56.429351+0000 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:56.428 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 1 Processor -- start 2024-09-11T05:07:56.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 1 -- start start 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient: init 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99d80a4030 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f99d80a4b20 0x7f99d80a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient(hunting): picked mon.noname-c con 0x7f99d80a4b20 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d813da40 0x7f99d813fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient(hunting): picked mon.noname-b con 0x7f99d813da40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99d80a2850 con 0x7f99d80a4030 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f99d80a30f0 con 0x7f99d80a4b20 2024-09-11T05:07:56.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f99d80a2b30 con 0x7f99d813da40 2024-09-11T05:07:56.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:56.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99df638640 10 monclient(hunting): authenticate will time out at 959.865340s 2024-09-11T05:07:56.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d813da40 0x7f99d813fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99dd3d4640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f99d80a4030 0x7f99d80a4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38468/0 (socket says 172.21.15.138:38468) 2024-09-11T05:07:56.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99dd3d4640 1 -- 172.21.15.138:0/666639405 learned_addr learned my addr 172.21.15.138:0/666639405 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:56.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient(hunting): get_auth_request con 0x7f99d813da40 auth_method 0 2024-09-11T05:07:56.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99dcbd3640 1 --2- 172.21.15.138:0/666639405 >> v2:172.21.15.190:3300/0 conn(0x7f99d80a4b20 0x7f99d80a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99dcbd3640 10 monclient(hunting): get_auth_request con 0x7f99d80a4b20 auth_method 0 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99dcbd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99dcbd3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99dcbd3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 1 -- 172.21.15.138:0/666639405 >> v2:172.21.15.190:3300/0 conn(0x7f99d80a4b20 msgr2=0x7f99d80a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 1 --2- 172.21.15.138:0/666639405 >> v2:172.21.15.190:3300/0 conn(0x7f99d80a4b20 0x7f99d80a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 1 -- 172.21.15.138:0/666639405 >> v1:172.21.15.138:6789/0 conn(0x7f99d80a4030 legacy=0x7f99d80a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient: _finish_hunting 0 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 1 monclient: found mon.noname-b 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:56.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:56.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.426+0000 7f99ddbd5640 1 -- 172.21.15.138:0/666639405 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d80a2e10 con 0x7f99d813da40 2024-09-11T05:07:56.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99ddbd5640 1 -- 172.21.15.138:0/666639405 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d813da40 msgr2=0x7f99d813fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:56.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99ddbd5640 1 --2- 172.21.15.138:0/666639405 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d813da40 0x7f99d813fe30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009d90 tx=0x7f99d001c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a03463eb9e491618 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 1 -- 172.21.15.138:0/666639405 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d00ac070 con 0x7f99d813da40 2024-09-11T05:07:56.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:56.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: dump: 2024-09-11T05:07:56.441 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: _finish_auth 0 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:56.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 1 -- 172.21.15.138:0/666639405 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99d00a6650 con 0x7f99d813da40 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 5 monclient: authenticate success, global_id 4328 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 4 set_mon_vals no callback set 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 1 -- 172.21.15.138:0/666639405 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d00a6a20 con 0x7f99d813da40 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99cffff640 10 monclient: dump: 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.443 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 10 monclient: shutdown 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 1 -- 172.21.15.138:0/666639405 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d813da40 msgr2=0x7f99d813fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 1 --2- 172.21.15.138:0/666639405 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d813da40 0x7f99d813fe30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009d90 tx=0x7f99d001c6e0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 1 -- 172.21.15.138:0/666639405 shutdown_connections 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 1 --2- 172.21.15.138:0/666639405 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d813da40 0x7f99d813fe30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 1 --2- 172.21.15.138:0/666639405 >> v2:172.21.15.190:3300/0 conn(0x7f99d80a4b20 0x7f99d80a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 1 -- 172.21.15.138:0/666639405 >> 172.21.15.138:0/666639405 conn(0x7f99d80ab940 msgr2=0x7f99d80a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 1 -- 172.21.15.138:0/666639405 shutdown_connections 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 1 -- 172.21.15.138:0/666639405 wait complete. 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 10 monclient: build_initial_monmap 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.430+0000 7f99df638640 10 monclient: monmap: 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:56.445 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 librados: starting msgr at 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 librados: starting objecter 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 Processor -- start 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 -- start start 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 librados: setting wanted keys 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 librados: calling monclient init 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient: init 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:07:56.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99d80a4030 0x7f99d8125c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient(hunting): picked mon.noname-a con 0x7f99d80a4030 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d80a4b20 0x7f99d8123b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient(hunting): picked mon.noname-c con 0x7f99d80a4b20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99d813da40 0x7f99d813cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient(hunting): picked mon.noname-b con 0x7f99d813da40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8143060 con 0x7f99d80a4030 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8142ee0 con 0x7f99d813da40 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f99d81431e0 con 0x7f99d80a4b20 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:56.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient(hunting): authenticate will time out at 959.872760s 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d80a4b20 0x7f99d8123b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99d80a4030 0x7f99d8125c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d80a4b20 0x7f99d8123b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58982/0 (socket says 172.21.15.138:58982) 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 1 -- 172.21.15.138:0/1245671639 learned_addr learned my addr 172.21.15.138:0/1245671639 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 10 monclient(hunting): get_auth_request con 0x7f99d80a4b20 auth_method 0 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient(hunting): get_auth_request con 0x7f99d80a4030 auth_method 0 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99ddbd5640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99d813da40 0x7f99d813cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99ddbd5640 10 monclient(hunting): get_auth_request con 0x7f99d813da40 auth_method 0 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99ddbd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99ddbd5640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99ddbd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-09-11T05:07:56.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 -- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d80a4b20 msgr2=0x7f99d8123b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d80a4b20 0x7f99d8123b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 -- 172.21.15.138:0/1245671639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99d813da40 msgr2=0x7f99d813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99d813da40 0x7f99d813cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient: _finish_hunting 0 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 monclient: found mon.noname-a 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 -- 172.21.15.138:0/1245671639 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d80a23d0 con 0x7f99d80a4030 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dcbd3640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d80a4b20 0x7f99d8123b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 -- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99d80a4030 msgr2=0x7f99d8125c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99dd3d4640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99d80a4030 0x7f99d8125c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f99d400c440 tx=0x7f99d400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 client.?.objecter ms_handle_connect 0x7f99d80a4030 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:56.450 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 1 -- 172.21.15.138:0/1245671639 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d4017070 con 0x7f99d80a4030 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: dump: 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.451 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 1 -- 172.21.15.138:0/1245671639 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99d40070d0 con 0x7f99d80a4030 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 5 monclient: authenticate success, global_id 4338 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient: _renew_subs 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d81250f0 con 0x7f99d80a4030 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 1 -- 172.21.15.138:0/1245671639 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d40074c0 con 0x7f99d80a4030 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cdffb640 10 monclient: dump: 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.452 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99cf7fe640 4 set_mon_vals no callback set 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient: _renew_subs 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d813d760 con 0x7f99d80a4030 2024-09-11T05:07:56.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.434+0000 7f99df638640 1 librados: init done 2024-09-11T05:07:56.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99df638640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99df638640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99df638640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d8126cc0 con 0x7f99d80a4030 2024-09-11T05:07:56.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99cdffb640 1 -- 172.21.15.138:0/1245671639 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f99d401b3b0 con 0x7f99d80a4030 2024-09-11T05:07:56.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99cdffb640 1 --2- 172.21.15.138:0/1245671639 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99bc03cea0 0x7f99bc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99cdffb640 1 -- 172.21.15.138:0/1245671639 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f99d404e9c0 con 0x7f99d80a4030 2024-09-11T05:07:56.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99cdffb640 10 client.4338.objecter ms_dispatch 0x7f99d8000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:07:56.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99cdffb640 3 client.4338.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:07:56.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99cdffb640 3 client.4338.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:07:56.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99cdffb640 20 client.4338.objecter dump_active .. 0 homeless 2024-09-11T05:07:56.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99dcbd3640 1 --2- 172.21.15.138:0/1245671639 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99bc03cea0 0x7f99bc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.438+0000 7f99dcbd3640 10 monclient: get_auth_request con 0x7f99bc03cea0 auth_method 0 2024-09-11T05:07:56.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.442+0000 7f99cdffb640 10 client.4338.objecter ms_handle_connect 0x7f99bc03cea0 2024-09-11T05:07:56.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.442+0000 7f99dcbd3640 1 --2- 172.21.15.138:0/1245671639 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99bc03cea0 0x7f99bc03f360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f99d813fc00 tx=0x7f99c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.442+0000 7f99cdffb640 1 -- 172.21.15.138:0/1245671639 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f99d401a360 con 0x7f99d80a4030 2024-09-11T05:07:56.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.442+0000 7f99cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.442+0000 7f99cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:56.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.598+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f99d81270d0 con 0x7f99bc03cea0 2024-09-11T05:07:56.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.598+0000 7f99cdffb640 1 -- 172.21.15.138:0/1245671639 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34153 (secure 0 0 0) 0x7f99d81270d0 con 0x7f99bc03cea0 2024-09-11T05:07:56.604 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:07:56.605 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 10 librados: watch_flush enter 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 10 librados: watch_flush exit 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 20 client.4338.objecter shutdown clearing up homeless session... 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 10 client.4338.objecter successfully canceled tick 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99bc03cea0 msgr2=0x7f99bc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 --2- 172.21.15.138:0/1245671639 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99bc03cea0 0x7f99bc03f360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f99d813fc00 tx=0x7f99c8008040 comp rx=0 tx=0).stop 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 10 monclient: shutdown 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99d80a4030 msgr2=0x7f99d8125c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99d80a4030 0x7f99d8125c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f99d400c440 tx=0x7f99d400c900 comp rx=0 tx=0).stop 2024-09-11T05:07:56.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 shutdown_connections 2024-09-11T05:07:56.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 --2- 172.21.15.138:0/1245671639 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99bc03cea0 0x7f99bc03f360 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99d813da40 0x7f99d813cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99d80a4b20 0x7f99d8123b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 --2- 172.21.15.138:0/1245671639 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99d80a4030 0x7f99d8125c50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 >> 172.21.15.138:0/1245671639 conn(0x7f99d80ab940 msgr2=0x7f99d813e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:56.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 shutdown_connections 2024-09-11T05:07:56.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 -- 172.21.15.138:0/1245671639 wait complete. 2024-09-11T05:07:56.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.602+0000 7f99df638640 1 librados: shutdown 2024-09-11T05:07:56.622 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-11T05:07:56.242895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163296,"kb_used_data":1088,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067744,"statfs":{"total":579820584960,"available":579653369856,"internally_reserved":0,"allocated":1114112,"data_stored":1375018,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.578865"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.536048+0000","last_change":"2024-09-11T05:07:50.536048+0000","last_active":"2024-09-11T05:07:50.536048+0000","last_peered":"2024-09-11T05:07:50.536048+0000","last_clean":"2024-09-11T05:07:50.536048+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:07:50.536048+0000","last_undegraded":"2024-09-11T05:07:50.536048+0000","last_fullsized":"2024-09-11T05:07:50.536048+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:33.527347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543840+0000","last_change":"2024-09-11T05:07:50.543840+0000","last_active":"2024-09-11T05:07:50.543840+0000","last_peered":"2024-09-11T05:07:50.543840+0000","last_clean":"2024-09-11T05:07:50.543840+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:07:50.543840+0000","last_undegraded":"2024-09-11T05:07:50.543840+0000","last_fullsized":"2024-09-11T05:07:50.543840+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:09.809820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.535722+0000","last_change":"2024-09-11T05:07:50.535722+0000","last_active":"2024-09-11T05:07:50.535722+0000","last_peered":"2024-09-11T05:07:50.535722+0000","last_clean":"2024-09-11T05:07:50.535722+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:07:50.535722+0000","last_undegraded":"2024-09-11T05:07:50.535722+0000","last_fullsized":"2024-09-11T05:07:50.535722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:07.310720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543876+0000","last_change":"2024-09-11T05:07:50.543876+0000","last_active":"2024-09-11T05:07:50.543876+0000","last_peered":"2024-09-11T05:07:50.543876+0000","last_clean":"2024-09-11T05:07:50.543876+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:07:50.543876+0000","last_undegraded":"2024-09-11T05:07:50.543876+0000","last_fullsized":"2024-09-11T05:07:50.543876+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:47:12.140333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.543322+0000","last_change":"2024-09-11T05:07:50.541650+0000","last_active":"2024-09-11T05:07:50.543322+0000","last_peered":"2024-09-11T05:07:50.543322+0000","last_clean":"2024-09-11T05:07:50.543322+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:07:50.543322+0000","last_undegraded":"2024-09-11T05:07:50.543322+0000","last_fullsized":"2024-09-11T05:07:50.543322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:47.451082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.533338+0000","last_change":"2024-09-11T05:07:50.533338+0000","last_active":"2024-09-11T05:07:50.533338+0000","last_peered":"2024-09-11T05:07:50.533338+0000","last_clean":"2024-09-11T05:07:50.533338+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:07:50.533338+0000","last_undegraded":"2024-09-11T05:07:50.533338+0000","last_fullsized":"2024-09-11T05:07:50.533338+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:04.573351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.535448+0000","last_change":"2024-09-11T05:07:50.535448+0000","last_active":"2024-09-11T05:07:50.535448+0000","last_peered":"2024-09-11T05:07:50.535448+0000","last_clean":"2024-09-11T05:07:50.535448+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:07:50.535448+0000","last_undegraded":"2024-09-11T05:07:50.535448+0000","last_fullsized":"2024-09-11T05:07:50.535448+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:57.570691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.540810+0000","last_change":"2024-09-11T05:07:50.540810+0000","last_active":"2024-09-11T05:07:50.540810+0000","last_peered":"2024-09-11T05:07:50.540810+0000","last_clean":"2024-09-11T05:07:50.540810+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:07:50.540810+0000","last_undegraded":"2024-09-11T05:07:50.540810+0000","last_fullsized":"2024-09-11T05:07:50.540810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:39.133232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T05:07:50.534426+0000","last_change":"2024-09-11T05:07:43.495116+0000","last_active":"2024-09-11T05:07:50.534426+0000","last_peered":"2024-09-11T05:07:50.534426+0000","last_clean":"2024-09-11T05:07:50.534426+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:07:50.534426+0000","last_undegraded":"2024-09-11T05:07:50.534426+0000","last_fullsized":"2024-09-11T05:07:50.534426+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_clean_scrub_stamp":"2024-09-11T05:07:41.703163+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:01.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":581782,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":581782,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:07:56.623 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-11T05:07:56.624 DEBUG:tasks.mon_thrash:Yielding 2024-09-11T05:07:56.624 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T05:07:56.651 INFO:tasks.workunit:Pulling workunits from ref 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T05:07:56.652 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T05:07:56.652 DEBUG:teuthology.orchestra.run.smithi190:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T05:07:56.652 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-11T05:07:56.653 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726031276, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-11T05:07:56.653 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:07:56.653 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:07:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:07:56.660 INFO:teuthology.orchestra.run.smithi190.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T05:07:56.661 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T05:07:56.706 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-11T05:07:56.706 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-11T05:07:56.754 INFO:tasks.workunit:timeout=3h 2024-09-11T05:07:56.755 INFO:tasks.workunit:cleanup=True 2024-09-11T05:07:56.755 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient: get_monmap_and_config 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient: build_initial_monmap 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient: monmap: 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:56.774760+0000 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:56.774760+0000 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:56.773 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.774 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:56.774 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:56.774 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:56.774 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:56.774 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:56.774 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 1 Processor -- start 2024-09-11T05:07:56.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 1 -- start start 2024-09-11T05:07:56.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient: init 2024-09-11T05:07:56.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:56.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-11T05:07:56.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdeb006e0e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:56.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdeb009eb30 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdeb009ef40 0x7fdeb013e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb009ef40 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb006e4c0 con 0x7fdeb009eb30 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb009f540 con 0x7fdeb006e0e0 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdeb009f820 con 0x7fdeb009ef40 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb75d5640 10 monclient(hunting): authenticate will time out at 960.210937s 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb4b70640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fdeb009eb30 0x7fdeb013a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38474/0 (socket says 172.21.15.138:38474) 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb4b70640 1 -- 172.21.15.138:0/3411303841 learned_addr learned my addr 172.21.15.138:0/3411303841 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 1 --2- 172.21.15.138:0/3411303841 >> v2:172.21.15.190:3300/0 conn(0x7fdeb009ef40 0x7fdeb013e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient(hunting): get_auth_request con 0x7fdeb009ef40 auth_method 0 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 1 -- 172.21.15.138:0/3411303841 >> v1:172.21.15.190:6789/0 conn(0x7fdeb006e0e0 legacy=0x7fdeb009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 1 -- 172.21.15.138:0/3411303841 >> v1:172.21.15.138:6789/0 conn(0x7fdeb009eb30 legacy=0x7fdeb013a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient: _finish_hunting 0 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 1 monclient: found mon.noname-c 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:56.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:07:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.770+0000 7fdeb5b72640 1 -- 172.21.15.138:0/3411303841 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb009fb00 con 0x7fdeb009ef40 2024-09-11T05:07:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 1 -- 172.21.15.138:0/3411303841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009ef40 msgr2=0x7fdeb013e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:56.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 1 --2- 172.21.15.138:0/3411303841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009ef40 0x7fdeb013e060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009d90 tx=0x7fdea801c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e6afb5c298d6f81 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 1 -- 172.21.15.138:0/3411303841 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea80b2070 con 0x7fdeb009ef40 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: dump: 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: _finish_auth 0 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: _check_auth_tickets 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 5 monclient: authenticate success, global_id 4306 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 1 -- 172.21.15.138:0/3411303841 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea80a5920 con 0x7fdeb009ef40 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 4 set_mon_vals no callback set 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 1 -- 172.21.15.138:0/3411303841 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea80a5cf0 con 0x7fdeb009ef40 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdea3fff640 10 monclient: dump: 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.782 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient: shutdown 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- 172.21.15.138:0/3411303841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009ef40 msgr2=0x7fdeb013e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 --2- 172.21.15.138:0/3411303841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009ef40 0x7fdeb013e060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009d90 tx=0x7fdea801c8c0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- 172.21.15.138:0/3411303841 shutdown_connections 2024-09-11T05:07:56.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 --2- 172.21.15.138:0/3411303841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009ef40 0x7fdeb013e060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- 172.21.15.138:0/3411303841 >> 172.21.15.138:0/3411303841 conn(0x7fdeb0063640 msgr2=0x7fdeb0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- 172.21.15.138:0/3411303841 shutdown_connections 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- 172.21.15.138:0/3411303841 wait complete. 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient: build_initial_monmap 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient: monmap: 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:56.784 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:56.785 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 librados: starting msgr at 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 librados: starting objecter 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 Processor -- start 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- start start 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 librados: setting wanted keys 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 librados: calling monclient init 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient: init 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb006e0e0 0x7fdeb01326a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient(hunting): picked mon.noname-a con 0x7fdeb006e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009eb30 0x7fdeb0132be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient(hunting): picked mon.noname-b con 0x7fdeb009eb30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:56.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb009ef40 0x7fdeb01ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb009ef40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb0140f40 con 0x7fdeb006e0e0 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb0140dc0 con 0x7fdeb009eb30 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb01410c0 con 0x7fdeb009ef40 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb75d5640 10 monclient(hunting): authenticate will time out at 960.218526s 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb006e0e0 0x7fdeb01326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb006e0e0 0x7fdeb01326a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46608/0 (socket says 172.21.15.138:46608) 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 -- 172.21.15.138:0/612415440 learned_addr learned my addr 172.21.15.138:0/612415440 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:56.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb009ef40 0x7fdeb01ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient(hunting): get_auth_request con 0x7fdeb006e0e0 auth_method 0 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 10 monclient(hunting): get_auth_request con 0x7fdeb009ef40 auth_method 0 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:56.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb4b70640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009eb30 0x7fdeb0132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb4b70640 10 monclient(hunting): get_auth_request con 0x7fdeb009eb30 auth_method 0 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb4b70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb4b70640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb4b70640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 -- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb009ef40 msgr2=0x7fdeb01ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb009ef40 0x7fdeb01ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 -- 172.21.15.138:0/612415440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009eb30 msgr2=0x7fdeb0132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009eb30 0x7fdeb0132be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient: _finish_hunting 0 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 monclient: found mon.noname-a 2024-09-11T05:07:56.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5371640 1 -- 172.21.15.138:0/612415440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb013d6b0 con 0x7fdeb006e0e0 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.778+0000 7fdeb5b72640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb009ef40 0x7fdeb01ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb5371640 1 -- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb006e0e0 msgr2=0x7fdeb01326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb5371640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb006e0e0 0x7fdeb01326a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdea400b410 tx=0x7fdea400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 client.?.objecter ms_handle_connect 0x7fdeb006e0e0 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 1 -- 172.21.15.138:0/612415440 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4015030 con 0x7fdeb006e0e0 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: dump: 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.793 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 1 -- 172.21.15.138:0/612415440 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea4004020 con 0x7fdeb006e0e0 2024-09-11T05:07:56.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 5 monclient: authenticate success, global_id 4341 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 10 monclient: _renew_subs 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb013d9a0 con 0x7fdeb006e0e0 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 1 -- 172.21.15.138:0/612415440 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4004410 con 0x7fdeb006e0e0 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 monclient: dump: 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:56.795 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 10 monclient: _renew_subs 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea37fe640 4 set_mon_vals no callback set 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb009e110 con 0x7fdeb006e0e0 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 1 librados: init done 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:56.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb009e420 con 0x7fdeb006e0e0 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 1 -- 172.21.15.138:0/612415440 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdea4022060 con 0x7fdeb006e0e0 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 1 --2- 172.21.15.138:0/612415440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9403cf00 0x7fde9403f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 1 -- 172.21.15.138:0/612415440 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7fdea404ed70 con 0x7fdeb006e0e0 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 10 client.4341.objecter ms_dispatch 0x7fdeb0000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 3 client.4341.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 3 client.4341.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdea1ffb640 20 client.4341.objecter dump_active .. 0 homeless 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb4b70640 1 --2- 172.21.15.138:0/612415440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9403cf00 0x7fde9403f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:56.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.782+0000 7fdeb4b70640 10 monclient: get_auth_request con 0x7fde9403cf00 auth_method 0 2024-09-11T05:07:56.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.786+0000 7fdea1ffb640 10 client.4341.objecter ms_handle_connect 0x7fde9403cf00 2024-09-11T05:07:56.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.786+0000 7fdeb4b70640 1 --2- 172.21.15.138:0/612415440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9403cf00 0x7fde9403f3c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdeb01cf510 tx=0x7fdeac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:56.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.786+0000 7fdea1ffb640 1 -- 172.21.15.138:0/612415440 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fdea4003dc0 con 0x7fdeb006e0e0 2024-09-11T05:07:56.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.786+0000 7fdea1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:56.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:56.786+0000 7fdea1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:56.804 INFO:tasks.workunit.client.0.smithi190.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T05:07:57.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.018+0000 7fdeb75d5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:07:57.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.018+0000 7fdeb75d5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:07:57.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.018+0000 7fdeb75d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.018+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdeb009dea0 con 0x7fdeb006e0e0 2024-09-11T05:07:57.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdea1ffb640 1 -- 172.21.15.138:0/612415440 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fdea4003980 con 0x7fdeb006e0e0 2024-09-11T05:07:57.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdea1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:07:57.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdea1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:57.024 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:07:57.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 10 librados: watch_flush enter 2024-09-11T05:07:57.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 10 librados: watch_flush exit 2024-09-11T05:07:57.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 20 client.4341.objecter shutdown clearing up homeless session... 2024-09-11T05:07:57.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 10 client.4341.objecter successfully canceled tick 2024-09-11T05:07:57.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9403cf00 msgr2=0x7fde9403f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 --2- 172.21.15.138:0/612415440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9403cf00 0x7fde9403f3c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdeb01cf510 tx=0x7fdeac009040 comp rx=0 tx=0).stop 2024-09-11T05:07:57.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 10 monclient: shutdown 2024-09-11T05:07:57.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb006e0e0 msgr2=0x7fdeb01326a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb006e0e0 0x7fdeb01326a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdea400b410 tx=0x7fdea400b8d0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 shutdown_connections 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 --2- 172.21.15.138:0/612415440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9403cf00 0x7fde9403f3c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb009ef40 0x7fdeb01ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb009eb30 0x7fdeb0132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 --2- 172.21.15.138:0/612415440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb006e0e0 0x7fdeb01326a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 >> 172.21.15.138:0/612415440 conn(0x7fdeb0063640 msgr2=0x7fdeb00a29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 shutdown_connections 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 -- 172.21.15.138:0/612415440 wait complete. 2024-09-11T05:07:57.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.022+0000 7fdeb75d5640 1 librados: shutdown 2024-09-11T05:07:57.042 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:07:57.043 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:07:57.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:07:57.043 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:07:57.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.154+0000 7f0b18663640 10 monclient: get_monmap_and_config 2024-09-11T05:07:57.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.154+0000 7f0b18663640 10 monclient: build_initial_monmap 2024-09-11T05:07:57.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.154+0000 7f0b18663640 10 monclient: monmap: 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:57.161348+0000 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:57.161348+0000 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:57.160 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 1 Processor -- start 2024-09-11T05:07:57.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 1 -- start start 2024-09-11T05:07:57.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient: init 2024-09-11T05:07:57.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:57.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0b100a32c0 0x7f0b1009e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient(hunting): picked mon.noname-a con 0x7f0b100a32c0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b1009e960 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0b1009ed80 0x7f0b10135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient(hunting): picked mon.noname-c con 0x7f0b1009ed80 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b100a1b50 con 0x7f0b1009e960 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f0b100a1920 con 0x7f0b100a32c0 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0b100a36a0 con 0x7f0b1009ed80 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b18663640 10 monclient(hunting): authenticate will time out at 960.597300s 2024-09-11T05:07:57.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0b100a32c0 0x7f0b1009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0b100a32c0 0x7f0b1009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46616/0 (socket says 172.21.15.138:46616) 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 1 -- 172.21.15.138:0/2277300490 learned_addr learned my addr 172.21.15.138:0/2277300490 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient(hunting): get_auth_request con 0x7f0b100a32c0 auth_method 0 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b16c00640 1 --2- 172.21.15.138:0/2277300490 >> v2:172.21.15.190:3300/0 conn(0x7f0b1009ed80 0x7f0b10135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 1 -- 172.21.15.138:0/2277300490 >> v2:172.21.15.190:3300/0 conn(0x7f0b1009ed80 msgr2=0x7f0b10135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 1 --2- 172.21.15.138:0/2277300490 >> v2:172.21.15.190:3300/0 conn(0x7f0b1009ed80 0x7f0b10135e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 1 -- 172.21.15.138:0/2277300490 >> v1:172.21.15.190:6789/0 conn(0x7f0b1009e960 legacy=0x7f0b101356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 1 monclient: found mon.noname-a 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.158+0000 7f0b163ff640 1 -- 172.21.15.138:0/2277300490 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1009f560 con 0x7f0b100a32c0 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b163ff640 1 -- 172.21.15.138:0/2277300490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b100a32c0 msgr2=0x7f0b1009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b163ff640 1 --2- 172.21.15.138:0/2277300490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b100a32c0 0x7f0b1009e420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b100a2000 tx=0x7f0b0c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e966531ad41e56b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 1 -- 172.21.15.138:0/2277300490 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0b0070 con 0x7f0b100a32c0 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: dump: 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.169 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: _finish_auth 0 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 5 monclient: authenticate success, global_id 4344 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 1 -- 172.21.15.138:0/2277300490 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b0c004230 con 0x7f0b100a32c0 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:57.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 1 -- 172.21.15.138:0/2277300490 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c003620 con 0x7f0b100a32c0 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 4 set_mon_vals no callback set 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b153fd640 10 monclient: dump: 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.171 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 10 monclient: shutdown 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 -- 172.21.15.138:0/2277300490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b100a32c0 msgr2=0x7f0b1009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 --2- 172.21.15.138:0/2277300490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b100a32c0 0x7f0b1009e420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b100a2000 tx=0x7f0b0c01c3b0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 -- 172.21.15.138:0/2277300490 shutdown_connections 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 --2- 172.21.15.138:0/2277300490 >> v2:172.21.15.190:3300/0 conn(0x7f0b1009ed80 0x7f0b10135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 --2- 172.21.15.138:0/2277300490 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b100a32c0 0x7f0b1009e420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 -- 172.21.15.138:0/2277300490 >> 172.21.15.138:0/2277300490 conn(0x7f0b10063640 msgr2=0x7f0b10063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 -- 172.21.15.138:0/2277300490 shutdown_connections 2024-09-11T05:07:57.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 -- 172.21.15.138:0/2277300490 wait complete. 2024-09-11T05:07:57.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 10 monclient: build_initial_monmap 2024-09-11T05:07:57.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 10 monclient: monmap: 2024-09-11T05:07:57.178 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:57.178 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:57.178 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:57.178 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:57.178 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 librados: starting msgr at 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 librados: starting objecter 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 Processor -- start 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 -- start start 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 librados: setting wanted keys 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 1 librados: calling monclient init 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.162+0000 7f0b18663640 10 monclient: init 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:57.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0b1009e960 0x7f0b101d0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): picked mon.noname-c con 0x7f0b1009e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1009ed80 0x7f0b101355a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): picked mon.noname-a con 0x7f0b1009ed80 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b100a32c0 0x7f0b10135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): picked mon.noname-b con 0x7f0b100a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b100a12d0 con 0x7f0b1009ed80 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1009f190 con 0x7f0b100a32c0 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0b100a6210 con 0x7f0b1009e960 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): authenticate will time out at 960.604213s 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0b1009e960 0x7f0b101d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0b1009e960 0x7f0b101d0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58998/0 (socket says 172.21.15.138:58998) 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 -- 172.21.15.138:0/4230857178 learned_addr learned my addr 172.21.15.138:0/4230857178 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient(hunting): get_auth_request con 0x7f0b1009e960 auth_method 0 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b15bfe640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1009ed80 0x7f0b101355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b15bfe640 10 monclient(hunting): get_auth_request con 0x7f0b1009ed80 auth_method 0 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b15bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b15bfe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b15bfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b16c00640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b100a32c0 0x7f0b10135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b100a32c0 msgr2=0x7f0b10135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b100a32c0 0x7f0b10135ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1009ed80 msgr2=0x7f0b101355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1009ed80 0x7f0b101355a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient: _finish_hunting 0 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 monclient: found mon.noname-c 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 -- 172.21.15.138:0/4230857178 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b10121980 con 0x7f0b1009e960 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0b1009e960 msgr2=0x7f0b101d0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b163ff640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0b1009e960 0x7f0b101d0510 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0ae040 tx=0x7f0b0c019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 client.?.objecter ms_handle_connect 0x7f0b1009e960 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0b0050 con 0x7f0b1009e960 2024-09-11T05:07:57.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: dump: 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b0c0ba070 con 0x7f0b1009e960 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 5 monclient: authenticate success, global_id 4337 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c0ab650 con 0x7f0b1009e960 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient: _renew_subs 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b10136020 con 0x7f0b1009e960 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b14bfc640 4 set_mon_vals no callback set 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:57.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b06ffd640 10 monclient: dump: 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient: _renew_subs 2024-09-11T05:07:57.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b101227c0 con 0x7f0b1009e960 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 librados: init done 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006f2e0 0x7f0b1006f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.166+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b100a1450 con 0x7f0b1006f2e0 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f0b0c0ab870 con 0x7f0b1009e960 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006f2e0 0x7f0b1006f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 10 monclient: get_auth_request con 0x7f0b1006f2e0 auth_method 0 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 --2- 172.21.15.138:0/4230857178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0af403cea0 0x7f0af403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 1 -- 172.21.15.138:0/4230857178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b1009e420 con 0x7f0b1006f2e0 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f0b0c0e95e0 con 0x7f0b1009e960 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 client.4337.objecter ms_dispatch 0x7f0b10000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 3 client.4337.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 3 client.4337.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 20 client.4337.objecter dump_active .. 0 homeless 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b16c00640 1 --2- 172.21.15.138:0/4230857178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0af403cea0 0x7f0af403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b16c00640 10 monclient: get_auth_request con 0x7f0af403cea0 auth_method 0 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 client.4337.objecter ms_handle_connect 0x7f0b1006f2e0 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 client.4337.objecter resend_mon_ops 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006f2e0 msgr2=0x7f0b1006f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b15bfe640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006f2e0 0x7f0b1006f6c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0b10071360 tx=0x7f0b0000e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00017070 con 0x7f0b1006f2e0 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 monclient: dump: 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.189 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0b00004020 con 0x7f0b1006f2e0 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006f2e0 msgr2=0x7f0b1006f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006f2e0 0x7f0b1006f6c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0b10071360 tx=0x7f0b0000e9f0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006f2e0 msgr2=0x7f0b1006f6c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006f2e0 0x7f0b1006f6c0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0b10071360 tx=0x7f0b0000e9f0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b16c00640 1 --2- 172.21.15.138:0/4230857178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0af403cea0 0x7f0af403f360 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b08008e70 tx=0x7f0b080083a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.170+0000 7f0b06ffd640 10 client.4337.objecter ms_handle_connect 0x7f0af403cea0 2024-09-11T05:07:57.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b18663640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:07:57.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b18663640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006fc00 0x7f0b10138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b18663640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0b10069640 con 0x7f0b1006fc00 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006fc00 0x7f0b10138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 10 monclient: get_auth_request con 0x7f0b1006fc00 auth_method 0 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 1 -- 172.21.15.138:0/4230857178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0b1006cca0 con 0x7f0b1006fc00 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006fc00 msgr2=0x7f0b10138920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b06ffd640 10 client.4337.objecter ms_handle_connect 0x7f0b1006fc00 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b06ffd640 10 client.4337.objecter resend_mon_ops 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b15bfe640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006fc00 0x7f0b10138920 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0b00018000 tx=0x7f0b00004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00017040 con 0x7f0b1006fc00 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.178+0000 7f0b06ffd640 10 monclient: dump: 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.194 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f0b00006030 con 0x7f0b1006fc00 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006fc00 msgr2=0x7f0b10138920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b06ffd640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006fc00 0x7f0b10138920 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0b00018000 tx=0x7f0b00004700 comp rx=0 tx=0).stop 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b06ffd640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006fc00 msgr2=0x7f0b10138920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b06ffd640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0b1006fc00 0x7f0b10138920 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0b00018000 tx=0x7f0b00004700 comp rx=0 tx=0).stop 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:07:57.195 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 8, 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 35951, 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 29, 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:07:57.196 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.197 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:07:57.199 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.200 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.201 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.202 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:07:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.206 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.207 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:07:57.208 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.208 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.208 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:07:57.208 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 10 librados: watch_flush enter 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 10 librados: watch_flush exit 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 20 client.4337.objecter shutdown clearing up homeless session... 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 10 client.4337.objecter successfully canceled tick 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0af403cea0 msgr2=0x7f0af403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 --2- 172.21.15.138:0/4230857178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0af403cea0 0x7f0af403f360 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b08008e70 tx=0x7f0b080083a0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 10 monclient: shutdown 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0b1009e960 msgr2=0x7f0b101d0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 --2- 172.21.15.138:0/4230857178 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0b1009e960 0x7f0b101d0510 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0ae040 tx=0x7f0b0c019150 comp rx=0 tx=0).stop 2024-09-11T05:07:57.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b16c00640 1 -- 172.21.15.138:0/4230857178 reap_dead start 2024-09-11T05:07:57.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b16c00640 1 -- 172.21.15.138:0/4230857178 reap_dead start 2024-09-11T05:07:57.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 shutdown_connections 2024-09-11T05:07:57.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 >> 172.21.15.138:0/4230857178 conn(0x7f0b10063640 msgr2=0x7f0b100a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 shutdown_connections 2024-09-11T05:07:57.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 -- 172.21.15.138:0/4230857178 wait complete. 2024-09-11T05:07:57.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.182+0000 7f0b18663640 1 librados: shutdown 2024-09-11T05:07:57.211 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:07:57.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:07:57.211 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:07:57.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient: get_monmap_and_config 2024-09-11T05:07:57.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient: build_initial_monmap 2024-09-11T05:07:57.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient: monmap: 2024-09-11T05:07:57.329 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:57.330 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.330 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:57.331173+0000 2024-09-11T05:07:57.330 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:57.331173+0000 2024-09-11T05:07:57.330 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:57.330 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.330 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:57.331 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:57.331 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:57.331 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:57.331 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:57.331 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 1 Processor -- start 2024-09-11T05:07:57.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 1 -- start start 2024-09-11T05:07:57.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient: init 2024-09-11T05:07:57.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:57.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-11T05:07:57.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07b009da20 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:57.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f07b00a2030 0x7f07b013b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient(hunting): picked mon.noname-a con 0x7f07b00a2030 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:57.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07b00a24a0 addr v1:172.21.15.138:6789/0 2024-09-11T05:07:57.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b0064590 con 0x7f07b00a24a0 2024-09-11T05:07:57.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b00a2c70 con 0x7f07b009da20 2024-09-11T05:07:57.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f07b009de00 con 0x7f07b00a2030 2024-09-11T05:07:57.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:57.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b8706640 10 monclient(hunting): authenticate will time out at 960.767206s 2024-09-11T05:07:57.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f07b00a2030 0x7f07b013b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f07b00a2030 0x7f07b013b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46660/0 (socket says 172.21.15.138:46660) 2024-09-11T05:07:57.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 1 -- 172.21.15.138:0/3133763584 learned_addr learned my addr 172.21.15.138:0/3133763584 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:57.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b6ca3640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f07b00a24a0 0x7f07b013da70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38480/0 (socket says 172.21.15.138:38480) 2024-09-11T05:07:57.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient(hunting): get_auth_request con 0x7f07b00a2030 auth_method 0 2024-09-11T05:07:57.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2024-09-11T05:07:57.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 1 -- 172.21.15.138:0/3133763584 >> v1:172.21.15.190:6789/0 conn(0x7f07b009da20 legacy=0x7f07b00a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:57.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 1 -- 172.21.15.138:0/3133763584 >> v1:172.21.15.138:6789/0 conn(0x7f07b00a24a0 legacy=0x7f07b013da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:57.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient: _finish_hunting 0 2024-09-11T05:07:57.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 1 monclient: found mon.noname-a 2024-09-11T05:07:57.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:57.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.326+0000 7f07b5ca1640 1 -- 172.21.15.138:0/3133763584 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b00a2f50 con 0x7f07b00a2030 2024-09-11T05:07:57.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.330+0000 7f07b5ca1640 1 -- 172.21.15.138:0/3133763584 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 msgr2=0x7f07b013b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.330+0000 7f07b5ca1640 1 --2- 172.21.15.138:0/3133763584 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 0x7f07b013b520 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009870 tx=0x7f07a001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10afccfc48b6c942 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 1 -- 172.21.15.138:0/3133763584 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00b1070 con 0x7f07b00a2030 2024-09-11T05:07:57.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: dump: 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.351 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: _finish_auth 0 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: _check_auth_tickets 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 1 -- 172.21.15.138:0/3133763584 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a00ac620 con 0x7f07b00a2030 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 5 monclient: authenticate success, global_id 4356 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 1 -- 172.21.15.138:0/3133763584 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00ac9f0 con 0x7f07b00a2030 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:57.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b54a0640 10 monclient: dump: 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 4 set_mon_vals no callback set 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient: shutdown 2024-09-11T05:07:57.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- 172.21.15.138:0/3133763584 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 msgr2=0x7f07b013b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 --2- 172.21.15.138:0/3133763584 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 0x7f07b013b520 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009870 tx=0x7f07a001c420 comp rx=0 tx=0).stop 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- 172.21.15.138:0/3133763584 shutdown_connections 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 --2- 172.21.15.138:0/3133763584 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 0x7f07b013b520 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- 172.21.15.138:0/3133763584 >> 172.21.15.138:0/3133763584 conn(0x7f07b0063640 msgr2=0x7f07b0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- 172.21.15.138:0/3133763584 shutdown_connections 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- 172.21.15.138:0/3133763584 wait complete. 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient: build_initial_monmap 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient: monmap: 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:57.354 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 librados: starting msgr at 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 librados: starting objecter 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 Processor -- start 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- start start 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 librados: setting wanted keys 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 librados: calling monclient init 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient: init 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:07:57.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009da20 0x7f07b013c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient(hunting): picked mon.noname-c con 0x7f07b009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 0x7f07b009e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient(hunting): picked mon.noname-a con 0x7f07b00a2030 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07b00a24a0 0x7f07b009e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient(hunting): picked mon.noname-b con 0x7f07b00a24a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0141d40 con 0x7f07b00a2030 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0141bc0 con 0x7f07b00a24a0 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f07b0141ec0 con 0x7f07b009da20 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b8706640 10 monclient(hunting): authenticate will time out at 960.774448s 2024-09-11T05:07:57.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 0x7f07b009e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b64a2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009da20 0x7f07b013c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 0x7f07b009e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46674/0 (socket says 172.21.15.138:46674) 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 -- 172.21.15.138:0/1333947358 learned_addr learned my addr 172.21.15.138:0/1333947358 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient(hunting): get_auth_request con 0x7f07b00a2030 auth_method 0 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b64a2640 10 monclient(hunting): get_auth_request con 0x7f07b009da20 auth_method 0 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b64a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b64a2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b64a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009da20 msgr2=0x7f07b013c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b6ca3640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07b00a24a0 0x7f07b009e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009da20 0x7f07b013c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07b00a24a0 msgr2=0x7f07b009e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07b00a24a0 0x7f07b009e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient: _finish_hunting 0 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 monclient: found mon.noname-a 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b5ca1640 1 -- 172.21.15.138:0/1333947358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b0130250 con 0x7f07b00a2030 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b6ca3640 10 monclient: get_auth_request con 0x7f07b00a24a0 auth_method 0 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.334+0000 7f07b6ca3640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07b00a24a0 0x7f07b009e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b5ca1640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 msgr2=0x7f07b009e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b5ca1640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 0x7f07b009e450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f07a001ccc0 tx=0x7f07a001ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 client.?.objecter ms_handle_connect 0x7f07b00a2030 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00bb020 con 0x7f07b00a2030 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: dump: 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.359 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 5 monclient: authenticate success, global_id 4359 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a00b7420 con 0x7f07b00a2030 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a00b1040 con 0x7f07b00a2030 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:57.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a7fff640 4 set_mon_vals no callback set 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 monclient: dump: 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 10 monclient: _renew_subs 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b013cc80 con 0x7f07b00a2030 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 10 monclient: _renew_subs 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b013b7c0 con 0x7f07b00a2030 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 1 librados: init done 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009eed0 0x7f07b0133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f07b00a19a0 con 0x7f07b009eed0 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009eed0 0x7f07b0133df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 10 monclient: get_auth_request con 0x7f07b009eed0 auth_method 0 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f07a00b4030 con 0x7f07b00a2030 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b64a2640 1 -- 172.21.15.138:0/1333947358 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07b009f2b0 con 0x7f07b009eed0 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 1 --2- 172.21.15.138:0/1333947358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f079403cef0 0x7f079403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f07a00e9410 con 0x7f07b00a2030 2024-09-11T05:07:57.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 10 client.4359.objecter ms_dispatch 0x7f07b0000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 3 client.4359.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 3 client.4359.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07a6ffd640 20 client.4359.objecter dump_active .. 0 homeless 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b6ca3640 1 --2- 172.21.15.138:0/1333947358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f079403cef0 0x7f079403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.338+0000 7f07b6ca3640 10 monclient: get_auth_request con 0x7f079403cef0 auth_method 0 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07b64a2640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009eed0 msgr2=0x7f07b0133df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 10 client.4359.objecter ms_handle_connect 0x7f07b009eed0 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 10 client.4359.objecter resend_mon_ops 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07b64a2640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009eed0 0x7f07b0133df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00fb10 tx=0x7f07ac005820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac018030 con 0x7f07b009eed0 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 10 monclient: dump: 2024-09-11T05:07:57.364 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f07ac00dd30 con 0x7f07b009eed0 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009eed0 msgr2=0x7f07b0133df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009eed0 0x7f07b0133df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00fb10 tx=0x7f07ac005820 comp rx=0 tx=0).stop 2024-09-11T05:07:57.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009eed0 msgr2=0x7f07b0133df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:07:57.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b009eed0 0x7f07b0133df0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07b6ca3640 1 --2- 172.21.15.138:0/1333947358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f079403cef0 0x7f079403f3b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f07a8007fd0 tx=0x7f07a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.342+0000 7f07a6ffd640 10 client.4359.objecter ms_handle_connect 0x7f079403cef0 2024-09-11T05:07:57.367 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b8706640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:07:57.367 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b8706640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b0134330 0x7f07b0134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.367 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b8706640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f07b006cca0 con 0x7f07b0134330 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b64a2640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b0134330 0x7f07b0134740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b64a2640 10 monclient: get_auth_request con 0x7f07b0134330 auth_method 0 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b64a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b64a2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.346+0000 7f07b64a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b64a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b64a2640 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b64a2640 1 -- 172.21.15.138:0/1333947358 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07b013d1a0 con 0x7f07b0134330 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b64a2640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b0134330 msgr2=0x7f07b0134740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b64a2640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b0134330 0x7f07b0134740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00fb10 tx=0x7f07ac003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 10 client.4359.objecter ms_handle_connect 0x7f07b0134330 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 10 client.4359.objecter resend_mon_ops 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac007440 con 0x7f07b0134330 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 10 monclient: dump: 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.369 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f07ac007440 con 0x7f07b0134330 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b0134330 msgr2=0x7f07b0134740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b0134330 0x7f07b0134740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00fb10 tx=0x7f07ac003a80 comp rx=0 tx=0).stop 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b0134330 msgr2=0x7f07b0134740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07a6ffd640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07b0134330 0x7f07b0134740 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00fb10 tx=0x7f07ac003a80 comp rx=0 tx=0).stop 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:07:57.370 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 8, 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 36123, 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 29, 2024-09-11T05:07:57.371 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:07:57.372 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:07:57.372 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:07:57.372 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.372 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.372 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.372 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.372 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.372 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.373 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:07:57.374 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.375 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:07:57.376 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.377 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.379 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.381 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.382 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.383 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.383 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:07:57.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.383 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.383 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:07:57.384 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:07:57.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 10 librados: watch_flush enter 2024-09-11T05:07:57.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 10 librados: watch_flush exit 2024-09-11T05:07:57.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 20 client.4359.objecter shutdown clearing up homeless session... 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 10 client.4359.objecter successfully canceled tick 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f079403cef0 msgr2=0x7f079403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 --2- 172.21.15.138:0/1333947358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f079403cef0 0x7f079403f3b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f07a8007fd0 tx=0x7f07a8009040 comp rx=0 tx=0).stop 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 10 monclient: shutdown 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 msgr2=0x7f07b009e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 --2- 172.21.15.138:0/1333947358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07b00a2030 0x7f07b009e450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f07a001ccc0 tx=0x7f07a001ac60 comp rx=0 tx=0).stop 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b6ca3640 1 -- 172.21.15.138:0/1333947358 reap_dead start 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b6ca3640 1 -- 172.21.15.138:0/1333947358 reap_dead start 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 shutdown_connections 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 >> 172.21.15.138:0/1333947358 conn(0x7f07b0063640 msgr2=0x7f07b0133360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 shutdown_connections 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 -- 172.21.15.138:0/1333947358 wait complete. 2024-09-11T05:07:57.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.350+0000 7f07b8706640 1 librados: shutdown 2024-09-11T05:07:57.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:07:57.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:07:57.387 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient: get_monmap_and_config 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient: build_initial_monmap 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient: monmap: 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:57.501741+0000 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:57.501741+0000 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:57.500 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:57.501 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:57.501 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:57.501 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:57.501 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 1 Processor -- start 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 1 -- start start 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient: init 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fec78070010 0x7fec78135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient(hunting): picked mon.noname-a con 0x7fec78070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fec7809e6e0 0x7fec781382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient(hunting): picked mon.noname-c con 0x7fec7809e6e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:07:57.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec7809eb10 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec780a8990 con 0x7fec7809eb10 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fec780703f0 con 0x7fec78070010 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fec7809cbf0 con 0x7fec7809e6e0 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7c987640 10 monclient(hunting): authenticate will time out at 960.937803s 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fec78070010 0x7fec78135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fec78070010 0x7fec78135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46690/0 (socket says 172.21.15.138:46690) 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 1 -- 172.21.15.138:0/3572132094 learned_addr learned my addr 172.21.15.138:0/3572132094 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:57.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient(hunting): get_auth_request con 0x7fec78070010 auth_method 0 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec75d9b640 1 --2- 172.21.15.138:0/3572132094 >> v2:172.21.15.190:3300/0 conn(0x7fec7809e6e0 0x7fec781382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec75d9b640 10 monclient(hunting): get_auth_request con 0x7fec7809e6e0 auth_method 0 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec75d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 1 -- 172.21.15.138:0/3572132094 >> v2:172.21.15.190:3300/0 conn(0x7fec7809e6e0 msgr2=0x7fec781382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:57.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 1 --2- 172.21.15.138:0/3572132094 >> v2:172.21.15.190:3300/0 conn(0x7fec7809e6e0 0x7fec781382c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 1 -- 172.21.15.138:0/3572132094 >> v1:172.21.15.190:6789/0 conn(0x7fec7809eb10 legacy=0x7fec78138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:57.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient: _finish_hunting 0 2024-09-11T05:07:57.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 1 monclient: found mon.noname-a 2024-09-11T05:07:57.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:57.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.498+0000 7fec7659c640 1 -- 172.21.15.138:0/3572132094 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7809ced0 con 0x7fec78070010 2024-09-11T05:07:57.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7659c640 1 -- 172.21.15.138:0/3572132094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec78070010 msgr2=0x7fec78135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7659c640 1 --2- 172.21.15.138:0/3572132094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec78070010 0x7fec78135450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fec7806efb0 tx=0x7fec7001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf4ec99d3e1f11e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 1 -- 172.21.15.138:0/3572132094 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec700b2070 con 0x7fec78070010 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: dump: 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.510 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: _finish_auth 0 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: _check_auth_tickets 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 1 -- 172.21.15.138:0/3572132094 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec700ad640 con 0x7fec78070010 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 5 monclient: authenticate success, global_id 4362 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 1 -- 172.21.15.138:0/3572132094 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec700ada10 con 0x7fec78070010 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 4 set_mon_vals no callback set 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:57.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7559a640 10 monclient: dump: 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 10 monclient: shutdown 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 1 -- 172.21.15.138:0/3572132094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec78070010 msgr2=0x7fec78135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 1 --2- 172.21.15.138:0/3572132094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec78070010 0x7fec78135450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fec7806efb0 tx=0x7fec7001c3b0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 1 -- 172.21.15.138:0/3572132094 shutdown_connections 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 1 --2- 172.21.15.138:0/3572132094 >> v2:172.21.15.190:3300/0 conn(0x7fec7809e6e0 0x7fec781382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 1 --2- 172.21.15.138:0/3572132094 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec78070010 0x7fec78135450 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 1 -- 172.21.15.138:0/3572132094 >> 172.21.15.138:0/3572132094 conn(0x7fec780ab830 msgr2=0x7fec780a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 1 -- 172.21.15.138:0/3572132094 shutdown_connections 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 1 -- 172.21.15.138:0/3572132094 wait complete. 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 10 monclient: build_initial_monmap 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.502+0000 7fec7c987640 10 monclient: monmap: 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:57.513 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 librados: starting msgr at 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 librados: starting objecter 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 Processor -- start 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 -- start start 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 librados: setting wanted keys 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 librados: calling monclient init 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient: init 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:07:57.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec78070010 0x7fec781d0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient(hunting): picked mon.noname-b con 0x7fec78070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fec7809e6e0 0x7fec781d0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient(hunting): picked mon.noname-c con 0x7fec7809e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec7809eb10 0x7fec781d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient(hunting): picked mon.noname-a con 0x7fec7809eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fec78142e80 con 0x7fec7809eb10 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec78139120 con 0x7fec78070010 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fec78143000 con 0x7fec7809e6e0 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 10 monclient(hunting): authenticate will time out at 960.944823s 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fec7809e6e0 0x7fec781d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec7809eb10 0x7fec781d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec7809eb10 0x7fec781d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46700/0 (socket says 172.21.15.138:46700) 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fec7809e6e0 0x7fec781d0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59028/0 (socket says 172.21.15.138:59028) 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 -- 172.21.15.138:0/3217854303 learned_addr learned my addr 172.21.15.138:0/3217854303 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient(hunting): get_auth_request con 0x7fec7809eb10 auth_method 0 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 10 monclient(hunting): get_auth_request con 0x7fec7809e6e0 auth_method 0 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec75d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7659c640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec78070010 0x7fec781d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fec7809e6e0 msgr2=0x7fec781d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fec7809e6e0 0x7fec781d0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec78070010 msgr2=0x7fec781d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec78070010 0x7fec781d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 monclient: found mon.noname-a 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:57.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 -- 172.21.15.138:0/3217854303 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec780a1d10 con 0x7fec7809eb10 2024-09-11T05:07:57.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7659c640 10 monclient: get_auth_request con 0x7fec78070010 auth_method 0 2024-09-11T05:07:57.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7659c640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec78070010 0x7fec781d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:07:57.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec7809eb10 msgr2=0x7fec781d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec76d9d640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec7809eb10 0x7fec781d0ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c780 tx=0x7fec6800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 client.?.objecter ms_handle_connect 0x7fec7809eb10 2024-09-11T05:07:57.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:57.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6800ac60 con 0x7fec7809eb10 2024-09-11T05:07:57.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: dump: 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: _finish_auth 0 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:07:57.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec7c987640 5 monclient: authenticate success, global_id 4365 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6800ae00 con 0x7fec7809eb10 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68004bb0 con 0x7fec7809eb10 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec74d99640 4 set_mon_vals no callback set 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.506+0000 7fec66ffd640 10 monclient: dump: 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.523 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 10 monclient: _renew_subs 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec780a2c00 con 0x7fec7809eb10 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 10 monclient: _renew_subs 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec780a4b80 con 0x7fec7809eb10 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 1 librados: init done 2024-09-11T05:07:57.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812e860 0x7fec7812ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec780a0250 con 0x7fec7812e860 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fec68005030 con 0x7fec7809eb10 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec66ffd640 1 --2- 172.21.15.138:0/3217854303 >> v2:172.21.15.190:6800/1562782189 conn(0x7fec5803cea0 0x7fec5803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812e860 0x7fec7812ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 10 monclient: get_auth_request con 0x7fec7812e860 auth_method 0 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7fec6804f500 con 0x7fec7809eb10 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec66ffd640 10 client.4365.objecter ms_dispatch 0x7fec78000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec66ffd640 3 client.4365.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:07:57.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec66ffd640 3 client.4365.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec66ffd640 20 client.4365.objecter dump_active .. 0 homeless 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec75d9b640 1 --2- 172.21.15.138:0/3217854303 >> v2:172.21.15.190:6800/1562782189 conn(0x7fec5803cea0 0x7fec5803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec75d9b640 10 monclient: get_auth_request con 0x7fec5803cea0 auth_method 0 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.510+0000 7fec7659c640 1 -- 172.21.15.138:0/3217854303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec78133680 con 0x7fec7812e860 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 10 client.4365.objecter ms_handle_connect 0x7fec7812e860 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 10 client.4365.objecter resend_mon_ops 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec7659c640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812e860 msgr2=0x7fec7812ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec7659c640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812e860 0x7fec7812ec40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fec78135200 tx=0x7fec70019150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 10 client.4365.objecter ms_handle_connect 0x7fec5803cea0 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec75d9b640 1 --2- 172.21.15.138:0/3217854303 >> v2:172.21.15.190:6800/1562782189 conn(0x7fec5803cea0 0x7fec5803f360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fec780a3a90 tx=0x7fec6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec700b2050 con 0x7fec7812e860 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 10 monclient: dump: 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fec700b3040 con 0x7fec7812e860 2024-09-11T05:07:57.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:57.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812e860 msgr2=0x7fec7812ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812e860 0x7fec7812ec40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fec78135200 tx=0x7fec70019150 comp rx=0 tx=0).stop 2024-09-11T05:07:57.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812e860 msgr2=0x7fec7812ec40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:07:57.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.514+0000 7fec66ffd640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812e860 0x7fec7812ec40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7c987640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7c987640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812f180 0x7fec7812fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7c987640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec78135450 con 0x7fec7812f180 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812f180 0x7fec7812fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 10 monclient: get_auth_request con 0x7fec7812f180 auth_method 0 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.518+0000 7fec7659c640 1 -- 172.21.15.138:0/3217854303 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec78069fa0 con 0x7fec7812f180 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7659c640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812f180 msgr2=0x7fec7812fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7659c640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812f180 0x7fec7812fa80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec7001cc50 tx=0x7fec700a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 10 client.4365.objecter ms_handle_connect 0x7fec7812f180 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 10 client.4365.objecter resend_mon_ops 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70003a30 con 0x7fec7812f180 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 10 monclient: dump: 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.531 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7fec70003a30 con 0x7fec7812f180 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812f180 msgr2=0x7fec7812fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812f180 0x7fec7812fa80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec7001cc50 tx=0x7fec700a7000 comp rx=0 tx=0).stop 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812f180 msgr2=0x7fec7812fa80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:07:57.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec66ffd640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec7812f180 0x7fec7812fa80 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec7001cc50 tx=0x7fec700a7000 comp rx=0 tx=0).stop 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 8, 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 36263, 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:07:57.533 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 30, 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.534 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.535 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:07:57.536 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:07:57.537 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.538 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:07:57.539 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.540 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:07:57.541 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:07:57.543 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:07:57.544 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:07:57.545 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 10 librados: watch_flush enter 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 10 librados: watch_flush exit 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 20 client.4365.objecter shutdown clearing up homeless session... 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 10 client.4365.objecter successfully canceled tick 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 >> v2:172.21.15.190:6800/1562782189 conn(0x7fec5803cea0 msgr2=0x7fec5803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 1 --2- 172.21.15.138:0/3217854303 >> v2:172.21.15.190:6800/1562782189 conn(0x7fec5803cea0 0x7fec5803f360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fec780a3a90 tx=0x7fec6c008040 comp rx=0 tx=0).stop 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 10 monclient: shutdown 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec7809eb10 msgr2=0x7fec781d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec76d9d640 1 -- 172.21.15.138:0/3217854303 reap_dead start 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.522+0000 7fec7c987640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec7809eb10 0x7fec781d0ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec6800c780 tx=0x7fec6800cc40 comp rx=0 tx=0).stop 2024-09-11T05:07:57.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.526+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 shutdown_connections 2024-09-11T05:07:57.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.526+0000 7fec7c987640 1 --2- 172.21.15.138:0/3217854303 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fec7809eb10 0x7fec781d0ed0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.526+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 >> 172.21.15.138:0/3217854303 conn(0x7fec780ab830 msgr2=0x7fec7806ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.526+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 shutdown_connections 2024-09-11T05:07:57.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.526+0000 7fec7c987640 1 -- 172.21.15.138:0/3217854303 wait complete. 2024-09-11T05:07:57.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.526+0000 7fec7c987640 1 librados: shutdown 2024-09-11T05:07:57.548 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:07:57.548 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T05:07:57.548 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T05:07:57.548 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T05:07:57.549 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T05:07:57.549 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T05:07:57.549 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:07:57.559 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T05:07:57.559 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:07:57.560 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:07:57.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.718+0000 7fe79d3cb640 10 monclient: get_monmap_and_config 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.718+0000 7fe79d3cb640 10 monclient: build_initial_monmap 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.718+0000 7fe79d3cb640 10 monclient: monmap: 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:07:57.724664+0000 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:07:57.724664+0000 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:07:57.723 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:07:57.724 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:07:57.724 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:07:57.724 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 1 Processor -- start 2024-09-11T05:07:57.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 1 -- start start 2024-09-11T05:07:57.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient: init 2024-09-11T05:07:57.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe79806e0e0 0x7fe798136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient(hunting): picked mon.noname-a con 0x7fe79806e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe798136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient(hunting): picked mon.noname-b con 0x7fe798136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7980a1920 addr v1:172.21.15.190:6789/0 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7980a3270 con 0x7fe7980a1920 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fe79806e4c0 con 0x7fe79806e0e0 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe7980a0af0 con 0x7fe798136170 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:57.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe79d3cb640 10 monclient(hunting): authenticate will time out at 961.160832s 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe796ffd640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7fe79806e0e0 msgr2=0x7fe798136720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe796ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe79806e0e0 0x7fe798136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe798136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe798136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44962/0 (socket says 172.21.15.138:44962) 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 1 -- 172.21.15.138:0/248444062 learned_addr learned my addr 172.21.15.138:0/248444062 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient(hunting): get_auth_request con 0x7fe798136170 auth_method 0 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-09-11T05:07:57.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 1 -- 172.21.15.138:0/248444062 >> v2:172.21.15.138:3300/0 conn(0x7fe79806e0e0 msgr2=0x7fe798136720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:07:57.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 1 --2- 172.21.15.138:0/248444062 >> v2:172.21.15.138:3300/0 conn(0x7fe79806e0e0 0x7fe798136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 1 -- 172.21.15.138:0/248444062 >> v1:172.21.15.190:6789/0 conn(0x7fe7980a1920 legacy=0x7fe7980a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:07:57.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient: _finish_hunting 0 2024-09-11T05:07:57.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 1 monclient: found mon.noname-b 2024-09-11T05:07:57.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:57.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.722+0000 7fe7967fc640 1 -- 172.21.15.138:0/248444062 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7980a2f90 con 0x7fe798136170 2024-09-11T05:07:57.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe7967fc640 1 -- 172.21.15.138:0/248444062 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 msgr2=0x7fe798136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe7967fc640 1 --2- 172.21.15.138:0/248444062 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe798136c60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe790009e50 tx=0x7fe79001c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e244ca471b6887c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 1 -- 172.21.15.138:0/248444062 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7900a9480 con 0x7fe798136170 2024-09-11T05:07:57.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:07:57.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: dump: 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: _finish_auth 0 2024-09-11T05:07:57.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 1 -- 172.21.15.138:0/248444062 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7900ae070 con 0x7fe798136170 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 5 monclient: authenticate success, global_id 4352 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient: get_monmap_and_config success 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 1 -- 172.21.15.138:0/248444062 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7900a9840 con 0x7fe798136170 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 4 set_mon_vals no callback set 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:57.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe795ffb640 10 monclient: dump: 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient: shutdown 2024-09-11T05:07:57.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 -- 172.21.15.138:0/248444062 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 msgr2=0x7fe798136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/248444062 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe798136c60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe790009e50 tx=0x7fe79001c730 comp rx=0 tx=0).stop 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 -- 172.21.15.138:0/248444062 shutdown_connections 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/248444062 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe798136c60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/248444062 >> v2:172.21.15.138:3300/0 conn(0x7fe79806e0e0 0x7fe798136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 -- 172.21.15.138:0/248444062 >> 172.21.15.138:0/248444062 conn(0x7fe798063640 msgr2=0x7fe798063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 -- 172.21.15.138:0/248444062 shutdown_connections 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 -- 172.21.15.138:0/248444062 wait complete. 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient: build_initial_monmap 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient: monmap: 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:07:57.737 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 librados: starting msgr at 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 librados: starting objecter 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 Processor -- start 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 -- start start 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 librados: setting wanted keys 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 librados: calling monclient init 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient: init 2024-09-11T05:07:57.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient: _reopen_session rank -1 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe79806e0e0 0x7fe79813ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient(hunting): picked mon.noname-b con 0x7fe79806e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe7980a1920 0x7fe79809e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 10 monclient(hunting): picked mon.noname-a con 0x7fe7980a1920 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.726+0000 7fe79d3cb640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe79809e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient(hunting): picked mon.noname-c con 0x7fe798136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe7981410a0 con 0x7fe7980a1920 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe798140f20 con 0x7fe79806e0e0 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe798141220 con 0x7fe798136170 2024-09-11T05:07:57.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient(hunting): _renew_subs 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient(hunting): authenticate will time out at 961.168032s 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7967fc640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe7980a1920 msgr2=0x7fe79809e450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7967fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe7980a1920 0x7fe79809e450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe79809e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe79809e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44970/0 (socket says 172.21.15.138:44970) 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 -- 172.21.15.138:0/4206704841 learned_addr learned my addr 172.21.15.138:0/4206704841 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient(hunting): get_auth_request con 0x7fe798136170 auth_method 0 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:07:57.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe796ffd640 1 --2- 172.21.15.138:0/4206704841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe79806e0e0 0x7fe79813ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe796ffd640 10 monclient(hunting): get_auth_request con 0x7fe79806e0e0 auth_method 0 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe796ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe796ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 -- 172.21.15.138:0/4206704841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe79806e0e0 msgr2=0x7fe79813ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 --2- 172.21.15.138:0/4206704841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe79806e0e0 0x7fe79813ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 -- 172.21.15.138:0/4206704841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe7980a1920 msgr2=0x7fe79809e450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 --2- 172.21.15.138:0/4206704841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe7980a1920 0x7fe79809e450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient: _finish_hunting 0 2024-09-11T05:07:57.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 monclient: found mon.noname-c 2024-09-11T05:07:57.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:07:57.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 -- 172.21.15.138:0/4206704841 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe798136f70 con 0x7fe798136170 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 -- 172.21.15.138:0/4206704841 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 msgr2=0x7fe79809e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7977fe640 1 --2- 172.21.15.138:0/4206704841 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe79809e990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c410 tx=0x7fe78800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 client.?.objecter ms_handle_connect 0x7fe798136170 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 1 -- 172.21.15.138:0/4206704841 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788017070 con 0x7fe798136170 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: dump: 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.744 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 1 -- 172.21.15.138:0/4206704841 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe788005230 con 0x7fe798136170 2024-09-11T05:07:57.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 5 monclient: authenticate success, global_id 4355 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient: _renew_subs 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe798072fd0 con 0x7fe798136170 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 1 -- 172.21.15.138:0/4206704841 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7880117a0 con 0x7fe798136170 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe77f7fe640 10 monclient: dump: 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:07:57.746 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe7957fa640 4 set_mon_vals no callback set 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient: _renew_subs 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79809e110 con 0x7fe798136170 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 1 librados: init done 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.730+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7980a1db0 con 0x7fe798136170 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe77f7fe640 1 -- 172.21.15.138:0/4206704841 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fe7880119c0 con 0x7fe798136170 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe77f7fe640 1 --2- 172.21.15.138:0/4206704841 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe78403ce50 0x7fe78403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe77f7fe640 1 -- 172.21.15.138:0/4206704841 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7fe78804e760 con 0x7fe798136170 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe77f7fe640 10 client.4355.objecter ms_dispatch 0x7fe798000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe77f7fe640 3 client.4355.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe77f7fe640 3 client.4355.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe77f7fe640 20 client.4355.objecter dump_active .. 0 homeless 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe796ffd640 1 --2- 172.21.15.138:0/4206704841 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe78403ce50 0x7fe78403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe796ffd640 10 monclient: get_auth_request con 0x7fe78403ce50 auth_method 0 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe77f7fe640 10 client.4355.objecter ms_handle_connect 0x7fe78403ce50 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.734+0000 7fe796ffd640 1 --2- 172.21.15.138:0/4206704841 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe78403ce50 0x7fe78403f310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe78c006fd0 tx=0x7fe78c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.738+0000 7fe77f7fe640 1 -- 172.21.15.138:0/4206704841 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe788004d40 con 0x7fe798136170 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.738+0000 7fe77f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:07:57.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.738+0000 7fe77f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:07:57.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.970+0000 7fe79d3cb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:07:57.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.970+0000 7fe79d3cb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:07:57.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.970+0000 7fe79d3cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:07:57.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.970+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe79806ccd0 con 0x7fe798136170 2024-09-11T05:07:57.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.970+0000 7fe77f7fe640 1 -- 172.21.15.138:0/4206704841 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fe7880192c0 con 0x7fe798136170 2024-09-11T05:07:57.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.970+0000 7fe77f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:07:57.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.970+0000 7fe77f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:07:57.975 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:07:57.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 10 librados: watch_flush enter 2024-09-11T05:07:57.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 10 librados: watch_flush exit 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 20 client.4355.objecter shutdown clearing up homeless session... 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 10 client.4355.objecter successfully canceled tick 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe78403ce50 msgr2=0x7fe78403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/4206704841 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe78403ce50 0x7fe78403f310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe78c006fd0 tx=0x7fe78c008040 comp rx=0 tx=0).stop 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 10 monclient: shutdown 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 msgr2=0x7fe79809e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/4206704841 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe79809e990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c410 tx=0x7fe78800c8d0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 shutdown_connections 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/4206704841 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe78403ce50 0x7fe78403f310 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/4206704841 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe798136170 0x7fe79809e990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/4206704841 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe7980a1920 0x7fe79809e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 --2- 172.21.15.138:0/4206704841 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe79806e0e0 0x7fe79813ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:07:57.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 >> 172.21.15.138:0/4206704841 conn(0x7fe798063640 msgr2=0x7fe798134600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:07:57.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 shutdown_connections 2024-09-11T05:07:57.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 -- 172.21.15.138:0/4206704841 wait complete. 2024-09-11T05:07:57.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:07:57.974+0000 7fe79d3cb640 1 librados: shutdown 2024-09-11T05:08:00.994 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:08:01.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient: get_monmap_and_config 2024-09-11T05:08:01.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient: build_initial_monmap 2024-09-11T05:08:01.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient: monmap: 2024-09-11T05:08:01.125 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:01.125 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:01.125 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:08:01.126943+0000 2024-09-11T05:08:01.125 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:08:01.126943+0000 2024-09-11T05:08:01.126 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:01.126 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:01.126 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:08:01.126 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:08:01.126 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:08:01.126 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:08:01.126 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:08:01.126 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:01.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 1 Processor -- start 2024-09-11T05:08:01.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 1 -- start start 2024-09-11T05:08:01.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient: init 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdda006e0e0 0x7fdda009e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient(hunting): picked mon.noname-c con 0x7fdda006e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdda009e960 0x7fdda009ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient(hunting): picked mon.noname-a con 0x7fdda009e960 addr v2:172.21.15.138:3300/0 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda01356f0 0x7fdda0135b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient(hunting): picked mon.noname-b con 0x7fdda01356f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fdda00a19b0 con 0x7fdda009e960 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdda009f820 con 0x7fdda006e0e0 2024-09-11T05:08:01.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdda006e4c0 con 0x7fdda01356f0 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda72a4640 10 monclient(hunting): authenticate will time out at 964.563037s 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda483f640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7fdda009e960 msgr2=0x7fdda009ed70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda483f640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdda009e960 0x7fdda009ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda01356f0 0x7fdda0135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda01356f0 0x7fdda0135b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44974/0 (socket says 172.21.15.138:44974) 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 -- 172.21.15.138:0/2260832321 learned_addr learned my addr 172.21.15.138:0/2260832321 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient(hunting): get_auth_request con 0x7fdda01356f0 auth_method 0 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:01.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5040640 1 --2- 172.21.15.138:0/2260832321 >> v2:172.21.15.190:3300/0 conn(0x7fdda006e0e0 0x7fdda009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 -- 172.21.15.138:0/2260832321 >> v2:172.21.15.190:3300/0 conn(0x7fdda006e0e0 msgr2=0x7fdda009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 --2- 172.21.15.138:0/2260832321 >> v2:172.21.15.190:3300/0 conn(0x7fdda006e0e0 0x7fdda009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 -- 172.21.15.138:0/2260832321 >> v2:172.21.15.138:3300/0 conn(0x7fdda009e960 msgr2=0x7fdda009ed70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 --2- 172.21.15.138:0/2260832321 >> v2:172.21.15.138:3300/0 conn(0x7fdda009e960 0x7fdda009ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient: _finish_hunting 0 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 monclient: found mon.noname-b 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:08:01.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.122+0000 7fdda5841640 1 -- 172.21.15.138:0/2260832321 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda009f540 con 0x7fdda01356f0 2024-09-11T05:08:01.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.126+0000 7fdda5841640 1 -- 172.21.15.138:0/2260832321 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda01356f0 msgr2=0x7fdda0135b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:01.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.126+0000 7fdda5841640 1 --2- 172.21.15.138:0/2260832321 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda01356f0 0x7fdda0135b00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdda0069420 tx=0x7fdd9801c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3bf52841c8acfd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:01.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.126+0000 7fdd97fff640 1 -- 172.21.15.138:0/2260832321 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd980a5640 con 0x7fdda01356f0 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.126+0000 7fdd97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 10 monclient: dump: 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:01.136 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 10 monclient: _finish_auth 0 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 10 monclient: _check_auth_tickets 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 5 monclient: authenticate success, global_id 4358 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 1 -- 172.21.15.138:0/2260832321 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd980a57e0 con 0x7fdda01356f0 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient: get_monmap_and_config success 2024-09-11T05:08:01.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 1 -- 172.21.15.138:0/2260832321 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd980a5bb0 con 0x7fdda01356f0 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 4 set_mon_vals no callback set 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdd97fff640 10 monclient: dump: 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:01.138 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient: shutdown 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- 172.21.15.138:0/2260832321 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda01356f0 msgr2=0x7fdda0135b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 --2- 172.21.15.138:0/2260832321 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda01356f0 0x7fdda0135b00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdda0069420 tx=0x7fdd9801c8a0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- 172.21.15.138:0/2260832321 shutdown_connections 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 --2- 172.21.15.138:0/2260832321 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda01356f0 0x7fdda0135b00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 --2- 172.21.15.138:0/2260832321 >> v2:172.21.15.138:3300/0 conn(0x7fdda009e960 0x7fdda009ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 --2- 172.21.15.138:0/2260832321 >> v2:172.21.15.190:3300/0 conn(0x7fdda006e0e0 0x7fdda009e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- 172.21.15.138:0/2260832321 >> 172.21.15.138:0/2260832321 conn(0x7fdda0063640 msgr2=0x7fdda0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- 172.21.15.138:0/2260832321 shutdown_connections 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- 172.21.15.138:0/2260832321 wait complete. 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient: build_initial_monmap 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient: monmap: 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:08:01.140 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 librados: starting msgr at 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 librados: starting objecter 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 Processor -- start 2024-09-11T05:08:01.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- start start 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 librados: setting wanted keys 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 librados: calling monclient init 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient: init 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda006e0e0 0x7fdda011d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient(hunting): picked mon.noname-c con 0x7fdda006e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdda009e960 0x7fdda011dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient(hunting): picked mon.noname-b con 0x7fdda009e960 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdda01356f0 0x7fdda011dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient(hunting): picked mon.noname-a con 0x7fdda01356f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:08:01.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdda00a1310 con 0x7fdda01356f0 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdda00a0950 con 0x7fdda009e960 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdda00a29c0 con 0x7fdda006e0e0 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda72a4640 10 monclient(hunting): authenticate will time out at 964.570287s 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5841640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdda01356f0 msgr2=0x7fdda011dff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda006e0e0 0x7fdda011d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5841640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdda01356f0 0x7fdda011dff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda006e0e0 0x7fdda011d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44980/0 (socket says 172.21.15.138:44980) 2024-09-11T05:08:01.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 -- 172.21.15.138:0/1775838293 learned_addr learned my addr 172.21.15.138:0/1775838293 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient(hunting): get_auth_request con 0x7fdda006e0e0 auth_method 0 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda483f640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdda009e960 0x7fdda011dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 -- 172.21.15.138:0/1775838293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdda009e960 msgr2=0x7fdda011dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdda009e960 0x7fdda011dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 -- 172.21.15.138:0/1775838293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdda01356f0 msgr2=0x7fdda011dff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:08:01.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdda01356f0 0x7fdda011dff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient: _finish_hunting 0 2024-09-11T05:08:01.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 monclient: found mon.noname-c 2024-09-11T05:08:01.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:01.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:08:01.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda5040640 1 -- 172.21.15.138:0/1775838293 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda0120070 con 0x7fdda006e0e0 2024-09-11T05:08:01.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda483f640 10 monclient: get_auth_request con 0x7fdda009e960 auth_method 0 2024-09-11T05:08:01.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.130+0000 7fdda483f640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdda009e960 0x7fdda011dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:08:01.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda5040640 1 -- 172.21.15.138:0/1775838293 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda006e0e0 msgr2=0x7fdda011d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:01.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda5040640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda006e0e0 0x7fdda011d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00c7f0 tx=0x7fdd9c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:01.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 client.?.objecter ms_handle_connect 0x7fdda006e0e0 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 1 -- 172.21.15.138:0/1775838293 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c015070 con 0x7fdda006e0e0 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: dump: 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:01.149 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: _finish_auth 0 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 5 monclient: authenticate success, global_id 4361 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 1 -- 172.21.15.138:0/1775838293 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9c004240 con 0x7fdda006e0e0 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 1 -- 172.21.15.138:0/1775838293 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c007830 con 0x7fdda006e0e0 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 10 monclient: _renew_subs 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda013c0c0 con 0x7fdda006e0e0 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:01.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd977fe640 4 set_mon_vals no callback set 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 monclient: dump: 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:08:01.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 10 monclient: _renew_subs 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda011f7d0 con 0x7fdda006e0e0 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 1 librados: init done 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda011f9d0 con 0x7fdda006e0e0 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 1 -- 172.21.15.138:0/1775838293 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdd9c007a50 con 0x7fdda006e0e0 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 1 --2- 172.21.15.138:0/1775838293 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd8403ce50 0x7fdd8403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 1 -- 172.21.15.138:0/1775838293 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7fdd9c04f280 con 0x7fdda006e0e0 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 10 client.4361.objecter ms_dispatch 0x7fdda0000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 3 client.4361.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 3 client.4361.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:08:01.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdd95ffb640 20 client.4361.objecter dump_active .. 0 homeless 2024-09-11T05:08:01.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda483f640 1 --2- 172.21.15.138:0/1775838293 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd8403ce50 0x7fdd8403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:01.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.134+0000 7fdda483f640 10 monclient: get_auth_request con 0x7fdd8403ce50 auth_method 0 2024-09-11T05:08:01.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.138+0000 7fdda483f640 1 --2- 172.21.15.138:0/1775838293 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd8403ce50 0x7fdd8403f310 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdda013c730 tx=0x7fdd90008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:01.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.138+0000 7fdd95ffb640 10 client.4361.objecter ms_handle_connect 0x7fdd8403ce50 2024-09-11T05:08:01.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.138+0000 7fdd95ffb640 1 -- 172.21.15.138:0/1775838293 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fdda011f9d0 con 0x7fdda006e0e0 2024-09-11T05:08:01.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.138+0000 7fdd95ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:01.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.138+0000 7fdd95ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:08:01.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.374+0000 7fdda72a4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:08:01.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.374+0000 7fdda72a4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:08:01.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.374+0000 7fdda72a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:01.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.374+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdda00a2d80 con 0x7fdda006e0e0 2024-09-11T05:08:01.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.374+0000 7fdd95ffb640 1 -- 172.21.15.138:0/1775838293 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fdd9c007d70 con 0x7fdda006e0e0 2024-09-11T05:08:01.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.374+0000 7fdd95ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:08:01.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.374+0000 7fdd95ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:08:01.378 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":34,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:08:01.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 10 librados: watch_flush enter 2024-09-11T05:08:01.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 10 librados: watch_flush exit 2024-09-11T05:08:01.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 20 client.4361.objecter shutdown clearing up homeless session... 2024-09-11T05:08:01.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 10 client.4361.objecter successfully canceled tick 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd8403ce50 msgr2=0x7fdd8403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 --2- 172.21.15.138:0/1775838293 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd8403ce50 0x7fdd8403f310 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdda013c730 tx=0x7fdd90008040 comp rx=0 tx=0).stop 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 10 monclient: shutdown 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda006e0e0 msgr2=0x7fdda011d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda006e0e0 0x7fdda011d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00c7f0 tx=0x7fdd9c00ccb0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 shutdown_connections 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 --2- 172.21.15.138:0/1775838293 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd8403ce50 0x7fdd8403f310 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdda01356f0 0x7fdda011dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdda009e960 0x7fdda011dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 --2- 172.21.15.138:0/1775838293 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdda006e0e0 0x7fdda011d570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 >> 172.21.15.138:0/1775838293 conn(0x7fdda0063640 msgr2=0x7fdda00a63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 shutdown_connections 2024-09-11T05:08:01.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 -- 172.21.15.138:0/1775838293 wait complete. 2024-09-11T05:08:01.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:01.378+0000 7fdda72a4640 1 librados: shutdown 2024-09-11T05:08:04.397 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:08:04.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient: get_monmap_and_config 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient: build_initial_monmap 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient: monmap: 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:08:04.527086+0000 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:08:04.527086+0000 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:08:04.526 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:08:04.527 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:04.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 1 Processor -- start 2024-09-11T05:08:04.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 1 -- start start 2024-09-11T05:08:04.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient: init 2024-09-11T05:08:04.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e6406e0e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f7e64136e30 0x7f7e6413b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient(hunting): picked mon.noname-a con 0x7f7e64136e30 addr v2:172.21.15.138:3300/0 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136170 0x7f7e6413da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient(hunting): picked mon.noname-b con 0x7f7e64136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e640a3270 con 0x7f7e6406e0e0 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f7e6406e4c0 con 0x7f7e64136e30 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7e640a0af0 con 0x7f7e64136170 2024-09-11T05:08:04.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e694aa640 10 monclient(hunting): authenticate will time out at 967.963052s 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e5bfff640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7f7e64136e30 msgr2=0x7f7e6413b4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e62ffd640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7f7e6406e0e0 legacy=0x7f7e64136720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e5bfff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f7e64136e30 0x7f7e6413b4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136170 0x7f7e6413da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136170 0x7f7e6413da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44992/0 (socket says 172.21.15.138:44992) 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 1 -- 172.21.15.138:0/1255661096 learned_addr learned my addr 172.21.15.138:0/1255661096 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient(hunting): get_auth_request con 0x7f7e64136170 auth_method 0 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:04.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 1 -- 172.21.15.138:0/1255661096 >> v2:172.21.15.138:3300/0 conn(0x7f7e64136e30 msgr2=0x7f7e6413b4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 1 --2- 172.21.15.138:0/1255661096 >> v2:172.21.15.138:3300/0 conn(0x7f7e64136e30 0x7f7e6413b4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 1 -- 172.21.15.138:0/1255661096 >> v1:172.21.15.138:6789/0 conn(0x7f7e6406e0e0 legacy=0x7f7e64136720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient: _finish_hunting 0 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 1 monclient: found mon.noname-b 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:08:04.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.522+0000 7f7e637fe640 1 -- 172.21.15.138:0/1255661096 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e640a2f90 con 0x7f7e64136170 2024-09-11T05:08:04.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 -- 172.21.15.138:0/1255661096 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136170 msgr2=0x7f7e6413da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:04.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 --2- 172.21.15.138:0/1255661096 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136170 0x7f7e6413da20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009e50 tx=0x7f7e5c01c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dc0d79b68149e46 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:04.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 1 -- 172.21.15.138:0/1255661096 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0a5510 con 0x7f7e64136170 2024-09-11T05:08:04.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:04.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:08:04.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: dump: 2024-09-11T05:08:04.532 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: _finish_auth 0 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: _check_auth_tickets 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 1 -- 172.21.15.138:0/1255661096 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e5c0a56b0 con 0x7f7e64136170 2024-09-11T05:08:04.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 1 -- 172.21.15.138:0/1255661096 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0a5a80 con 0x7f7e64136170 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e627fc640 10 monclient: dump: 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:04.534 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:04.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 5 monclient: authenticate success, global_id 4364 2024-09-11T05:08:04.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient: get_monmap_and_config success 2024-09-11T05:08:04.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 4 set_mon_vals no callback set 2024-09-11T05:08:04.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient: shutdown 2024-09-11T05:08:04.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:04.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- 172.21.15.138:0/1255661096 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136170 msgr2=0x7f7e6413da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:04.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 --2- 172.21.15.138:0/1255661096 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136170 0x7f7e6413da20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009e50 tx=0x7f7e5c01c7c0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- 172.21.15.138:0/1255661096 shutdown_connections 2024-09-11T05:08:04.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 --2- 172.21.15.138:0/1255661096 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136170 0x7f7e6413da20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 --2- 172.21.15.138:0/1255661096 >> v2:172.21.15.138:3300/0 conn(0x7f7e64136e30 0x7f7e6413b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- 172.21.15.138:0/1255661096 >> 172.21.15.138:0/1255661096 conn(0x7f7e64063640 msgr2=0x7f7e64063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- 172.21.15.138:0/1255661096 shutdown_connections 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- 172.21.15.138:0/1255661096 wait complete. 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient: build_initial_monmap 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient: monmap: 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:08:04.541 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 librados: starting msgr at 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 librados: starting objecter 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 Processor -- start 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- start start 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 librados: setting wanted keys 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 librados: calling monclient init 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient: init 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e6406e0e0 0x7f7e6413d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient(hunting): picked mon.noname-b con 0x7f7e6406e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:08:04.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e64136170 0x7f7e6413d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient(hunting): picked mon.noname-a con 0x7f7e64136170 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136e30 0x7f7e64137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient(hunting): picked mon.noname-c con 0x7f7e64136e30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7e64140b10 con 0x7f7e64136170 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e64140990 con 0x7f7e6406e0e0 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7e64140c90 con 0x7f7e64136e30 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e694aa640 10 monclient(hunting): authenticate will time out at 967.970899s 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e5bfff640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e64136170 msgr2=0x7f7e6413d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e5bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e64136170 0x7f7e6413d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:04.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136e30 0x7f7e64137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136e30 0x7f7e64137210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44994/0 (socket says 172.21.15.138:44994) 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 -- 172.21.15.138:0/3220447877 learned_addr learned my addr 172.21.15.138:0/3220447877 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient(hunting): get_auth_request con 0x7f7e64136e30 auth_method 0 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e62ffd640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e6406e0e0 0x7f7e6413d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 -- 172.21.15.138:0/3220447877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e6406e0e0 msgr2=0x7f7e6413d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e6406e0e0 0x7f7e6413d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 -- 172.21.15.138:0/3220447877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e64136170 msgr2=0x7f7e6413d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e64136170 0x7f7e6413d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient: _finish_hunting 0 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 monclient: found mon.noname-c 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e637fe640 1 -- 172.21.15.138:0/3220447877 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6413a330 con 0x7f7e64136e30 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e62ffd640 10 monclient: get_auth_request con 0x7f7e6406e0e0 auth_method 0 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.530+0000 7f7e62ffd640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e6406e0e0 0x7f7e6413d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e637fe640 1 -- 172.21.15.138:0/3220447877 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136e30 msgr2=0x7f7e64137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e637fe640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136e30 0x7f7e64137210 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009e50 tx=0x7f7e5c019560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 client.?.objecter ms_handle_connect 0x7f7e64136e30 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:08:04.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 1 -- 172.21.15.138:0/3220447877 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0bc020 con 0x7f7e64136e30 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: dump: 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 1 -- 172.21.15.138:0/3220447877 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e5c0b8a90 con 0x7f7e64136e30 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 5 monclient: authenticate success, global_id 4367 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 10 monclient: _renew_subs 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6413a610 con 0x7f7e64136e30 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 1 -- 172.21.15.138:0/3220447877 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0a5510 con 0x7f7e64136e30 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 monclient: dump: 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e61ffb640 4 set_mon_vals no callback set 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 10 monclient: _renew_subs 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e641388b0 con 0x7f7e64136e30 2024-09-11T05:08:04.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 1 librados: init done 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e64136720 con 0x7f7e64136e30 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 1 -- 172.21.15.138:0/3220447877 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f7e5c0b6440 con 0x7f7e64136e30 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 1 --2- 172.21.15.138:0/3220447877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e4403ce50 0x7f7e4403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 1 -- 172.21.15.138:0/3220447877 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f7e5c0ea720 con 0x7f7e64136e30 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 10 client.4367.objecter ms_dispatch 0x7f7e64000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 3 client.4367.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 3 client.4367.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e5b7fe640 20 client.4367.objecter dump_active .. 0 homeless 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e62ffd640 1 --2- 172.21.15.138:0/3220447877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e4403ce50 0x7f7e4403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.534+0000 7f7e62ffd640 10 monclient: get_auth_request con 0x7f7e4403ce50 auth_method 0 2024-09-11T05:08:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.538+0000 7f7e5b7fe640 10 client.4367.objecter ms_handle_connect 0x7f7e4403ce50 2024-09-11T05:08:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.538+0000 7f7e62ffd640 1 --2- 172.21.15.138:0/3220447877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e4403ce50 0x7f7e4403f310 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7e54003990 tx=0x7f7e54008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.538+0000 7f7e5b7fe640 1 -- 172.21.15.138:0/3220447877 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f7e5c0c6030 con 0x7f7e64136e30 2024-09-11T05:08:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.538+0000 7f7e5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.538+0000 7f7e5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:08:04.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.758+0000 7f7e694aa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:08:04.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.758+0000 7f7e694aa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:08:04.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.758+0000 7f7e694aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:04.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.758+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7e6413a980 con 0x7f7e64136e30 2024-09-11T05:08:04.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e5b7fe640 1 -- 172.21.15.138:0/3220447877 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f7e640a3270 con 0x7f7e64136e30 2024-09-11T05:08:04.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:08:04.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e5b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:08:04.765 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":37,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:08:04.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 10 librados: watch_flush enter 2024-09-11T05:08:04.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 10 librados: watch_flush exit 2024-09-11T05:08:04.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 20 client.4367.objecter shutdown clearing up homeless session... 2024-09-11T05:08:04.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 10 client.4367.objecter successfully canceled tick 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e4403ce50 msgr2=0x7f7e4403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 --2- 172.21.15.138:0/3220447877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e4403ce50 0x7f7e4403f310 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7e54003990 tx=0x7f7e54008040 comp rx=0 tx=0).stop 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 10 monclient: shutdown 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136e30 msgr2=0x7f7e64137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136e30 0x7f7e64137210 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009e50 tx=0x7f7e5c019560 comp rx=0 tx=0).stop 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 shutdown_connections 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 --2- 172.21.15.138:0/3220447877 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e4403ce50 0x7f7e4403f310 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e64136e30 0x7f7e64137210 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e64136170 0x7f7e6413d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 --2- 172.21.15.138:0/3220447877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e6406e0e0 0x7f7e6413d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 >> 172.21.15.138:0/3220447877 conn(0x7f7e64063640 msgr2=0x7f7e64132b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:04.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 shutdown_connections 2024-09-11T05:08:04.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 -- 172.21.15.138:0/3220447877 wait complete. 2024-09-11T05:08:04.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:04.762+0000 7f7e694aa640 1 librados: shutdown 2024-09-11T05:08:07.783 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:08:07.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient: get_monmap_and_config 2024-09-11T05:08:07.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient: build_initial_monmap 2024-09-11T05:08:07.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient: monmap: 2024-09-11T05:08:07.913 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:08:07.915220+0000 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:08:07.915220+0000 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:08:07.914 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:07.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 1 Processor -- start 2024-09-11T05:08:07.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 1 -- start start 2024-09-11T05:08:07.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient: init 2024-09-11T05:08:07.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:07.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T05:08:07.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46840a5b40 addr v1:172.21.15.190:6789/0 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient(hunting): picked mon.noname-b con 0x7f46840a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f468409d840 addr v1:172.21.15.138:6789/0 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f468409d490 con 0x7f468409d840 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4684070580 con 0x7f46840a5b40 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f468409e110 con 0x7f46840a7130 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f468a231640 10 monclient(hunting): authenticate will time out at 971.351421s 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4683fff640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7f468409d840 legacy=0x7f4684140090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:08:07.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:39418/0 (socket says 172.21.15.138:39418) 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 learned_addr learned my addr 172.21.15.138:0/520508982 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f46840a7130 auth_method 0 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:07.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.910+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:07.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:07.914+0000 7f46837fe640 1 -- 172.21.15.138:0/520508982 >> v1:172.21.15.190:6789/0 conn(0x7f46840a5b40 legacy=0x7f46840a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:08.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4683fff640 1 -- 172.21.15.138:0/520508982 >> v1:172.21.15.138:6789/0 conn(0x7f468409d840 legacy=0x7f4684140090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:08:08.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:08.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f46840a7130 auth_method 0 2024-09-11T05:08:08.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:08.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:08.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:08:08.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:08:08.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:08:08.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:08.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.114+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:08:08.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4683fff640 1 -- 172.21.15.138:0/520508982 >> v1:172.21.15.138:6789/0 conn(0x7f468409d840 legacy=0x7f4684140090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:08:08.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:08.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f46840a7130 auth_method 0 2024-09-11T05:08:08.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:08.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:08.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:08:08.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:08:08.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:08:08.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:08.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:08.518+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4683fff640 1 -- 172.21.15.138:0/520508982 >> v1:172.21.15.138:6789/0 conn(0x7f468409d840 legacy=0x7f4684140090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f46840a7130 auth_method 0 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:09.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:09.322+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:08:10.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient(hunting): tick 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 monclient(hunting): continuing hunt 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 msgr2=0x7f468413db40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f46840a7130 0x7f468413db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 -- 172.21.15.138:0/520508982 >> v1:172.21.15.190:6789/0 conn(0x7f46840a5b40 legacy=0x7f46840a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 -- 172.21.15.138:0/520508982 >> v1:172.21.15.138:6789/0 conn(0x7f468409d840 legacy=0x7f4684140090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient: _add_conns ranks=[3,4] 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 0x7f4670005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient(hunting): picked mon.noname-a con 0x7f4670003780 addr v2:172.21.15.138:3300/0 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:10.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient(hunting): picked mon.noname-c con 0x7f46700062d0 addr v2:172.21.15.190:3300/0 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 -- 172.21.15.138:0/520508982 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f468409e110 con 0x7f4670003780 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 1 -- 172.21.15.138:0/520508982 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f4670008e30 con 0x7f46700062d0 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46817fa640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46837fe640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 msgr2=0x7f4670005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.910+0000 7f46837fe640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 0x7f4670005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f46700062d0 auth_method 0 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:10.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:10.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 msgr2=0x7f4670008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:08:10.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 msgr2=0x7f4670008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:08:10.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:10.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:10.914+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:11.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.114+0000 7f46837fe640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 msgr2=0x7f4670005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:11.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.114+0000 7f46837fe640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 0x7f4670005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f46700062d0 auth_method 0 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 msgr2=0x7f4670008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 msgr2=0x7f4670008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:11.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.118+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:08:11.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.518+0000 7f46837fe640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 msgr2=0x7f4670005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:11.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.518+0000 7f46837fe640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 0x7f4670005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:08:11.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:11.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f46700062d0 auth_method 0 2024-09-11T05:08:11.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:11.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:11.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:08:11.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 msgr2=0x7f4670008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:08:11.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 msgr2=0x7f4670008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:08:11.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:11.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:11.522+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:08:12.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.322+0000 7f46837fe640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 msgr2=0x7f4670005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:12.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.322+0000 7f46837fe640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 0x7f4670005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:08:12.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.190:3300/0 conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f46700062d0 auth_method 0 2024-09-11T05:08:12.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:08:12.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2024-09-11T05:08:12.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 msgr2=0x7f4670005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:12.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> v2:172.21.15.138:3300/0 conn(0x7f4670003780 0x7f4670005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:08:12.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 1 monclient: found mon.noname-c 2024-09-11T05:08:12.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:12.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:08:12.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.326+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468409cb80 con 0x7f46700062d0 2024-09-11T05:08:12.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f4682ffd640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46700062d0 msgr2=0x7f4670008770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f4682ffd640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46700062d0 0x7f4670008770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46700062a0 tx=0x7f467801de40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6db09f8416b060ea server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 1 -- 172.21.15.138:0/520508982 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46780ac070 con 0x7f46700062d0 2024-09-11T05:08:12.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:12.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: dump: 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: _finish_auth 0 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: _check_auth_tickets 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:12.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 1 -- 172.21.15.138:0/520508982 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46780aa070 con 0x7f46700062d0 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 5 monclient: authenticate success, global_id 4324 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 1 -- 172.21.15.138:0/520508982 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467800f040 con 0x7f46700062d0 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 10 monclient: get_monmap_and_config success 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 4 set_mon_vals no callback set 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f46827fc640 10 monclient: dump: 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.339 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.340 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.340 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.340 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.340 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 10 monclient: shutdown 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 1 -- 172.21.15.138:0/520508982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46700062d0 msgr2=0x7f4670008770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46700062d0 0x7f4670008770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46700062a0 tx=0x7f467801de40 comp rx=0 tx=0).stop 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f4683fff640 1 -- 172.21.15.138:0/520508982 reap_dead start 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 1 -- 172.21.15.138:0/520508982 shutdown_connections 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 1 --2- 172.21.15.138:0/520508982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46700062d0 0x7f4670008770 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 1 -- 172.21.15.138:0/520508982 >> 172.21.15.138:0/520508982 conn(0x7f46840aba10 msgr2=0x7f46840a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 1 -- 172.21.15.138:0/520508982 shutdown_connections 2024-09-11T05:08:12.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.334+0000 7f468a231640 1 -- 172.21.15.138:0/520508982 wait complete. 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient: build_initial_monmap 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient: monmap: 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 librados: starting msgr at 2024-09-11T05:08:12.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 librados: starting objecter 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 Processor -- start 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 -- start start 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 librados: setting wanted keys 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 librados: calling monclient init 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient: init 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f46700062d0 0x7f4684138560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient(hunting): picked mon.noname-a con 0x7f46700062d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:08:12.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684138aa0 0x7f468413d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient(hunting): picked mon.noname-b con 0x7f4684138aa0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f468413dbc0 0x7f468413e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient(hunting): picked mon.noname-c con 0x7f468413dbc0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4684144360 con 0x7f46700062d0 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f46841441e0 con 0x7f4684138aa0 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f46841444e0 con 0x7f468413dbc0 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f468a231640 10 monclient(hunting): authenticate will time out at 975.777893s 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f46837fe640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f46700062d0 msgr2=0x7f4684138560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f46837fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f46700062d0 0x7f4684138560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:12.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f468413dbc0 0x7f468413e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684138aa0 0x7f468413d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f468413dbc0 0x7f468413e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:39434/0 (socket says 172.21.15.138:39434) 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 -- 172.21.15.138:0/1107858514 learned_addr learned my addr 172.21.15.138:0/1107858514 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient(hunting): get_auth_request con 0x7f468413dbc0 auth_method 0 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684138aa0 0x7f468413d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.138:38448/0 (socket says 172.21.15.138:38448) 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 10 monclient(hunting): get_auth_request con 0x7f4684138aa0 auth_method 0 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4682ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 -- 172.21.15.138:0/1107858514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684138aa0 msgr2=0x7f468413d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 --2- 172.21.15.138:0/1107858514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684138aa0 0x7f468413d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 -- 172.21.15.138:0/1107858514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f46700062d0 msgr2=0x7f4684138560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:08:12.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 --2- 172.21.15.138:0/1107858514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f46700062d0 0x7f4684138560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient: _finish_hunting 0 2024-09-11T05:08:12.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 monclient: found mon.noname-c 2024-09-11T05:08:12.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:12.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:08:12.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.338+0000 7f4683fff640 1 -- 172.21.15.138:0/1107858514 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4684121d50 con 0x7f468413dbc0 2024-09-11T05:08:12.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f4683fff640 1 -- 172.21.15.138:0/1107858514 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f468413dbc0 msgr2=0x7f468413e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f4683fff640 1 --2- 172.21.15.138:0/1107858514 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f468413dbc0 0x7f468413e040 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f468409dc90 tx=0x7f467c00be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 client.?.objecter ms_handle_connect 0x7f468413dbc0 2024-09-11T05:08:12.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:08:12.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 1 -- 172.21.15.138:0/1107858514 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c016430 con 0x7f468413dbc0 2024-09-11T05:08:12.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: dump: 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.353 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: _finish_auth 0 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 1 -- 172.21.15.138:0/1107858514 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f467c010030 con 0x7f468413dbc0 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 1 -- 172.21.15.138:0/1107858514 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c016830 con 0x7f468413dbc0 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 monclient: dump: 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.354 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 5 monclient: authenticate success, global_id 4370 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 10 monclient: _renew_subs 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f46817fa640 4 set_mon_vals no callback set 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4684122030 con 0x7f468413dbc0 2024-09-11T05:08:12.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 10 monclient: _renew_subs 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f468413fc00 con 0x7f468413dbc0 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 1 librados: init done 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 1 -- 172.21.15.138:0/1107858514 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f467c0153e0 con 0x7f468413dbc0 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46840a5f20 con 0x7f468413dbc0 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 1 --2- 172.21.15.138:0/1107858514 >> v2:172.21.15.190:6800/1562782189 conn(0x7f467003dcb0 0x7f4670040170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 1 -- 172.21.15.138:0/1107858514 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f467c04f360 con 0x7f468413dbc0 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 10 client.4370.objecter ms_dispatch 0x7f4684000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 3 client.4370.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:08:12.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 3 client.4370.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:08:12.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f466bfff640 20 client.4370.objecter dump_active .. 0 homeless 2024-09-11T05:08:12.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f46837fe640 1 --2- 172.21.15.138:0/1107858514 >> v2:172.21.15.190:6800/1562782189 conn(0x7f467003dcb0 0x7f4670040170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.342+0000 7f46837fe640 10 monclient: get_auth_request con 0x7f467003dcb0 auth_method 0 2024-09-11T05:08:12.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.346+0000 7f46837fe640 1 --2- 172.21.15.138:0/1107858514 >> v2:172.21.15.190:6800/1562782189 conn(0x7f467003dcb0 0x7f4670040170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4674001500 tx=0x7f4674007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.346+0000 7f466bfff640 10 client.4370.objecter ms_handle_connect 0x7f467003dcb0 2024-09-11T05:08:12.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.346+0000 7f466bfff640 1 -- 172.21.15.138:0/1107858514 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f467c04e240 con 0x7f468413dbc0 2024-09-11T05:08:12.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.346+0000 7f466bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:12.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.346+0000 7f466bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:08:12.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:08:12.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:08:12.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:12.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f46841407d0 con 0x7f468413dbc0 2024-09-11T05:08:12.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f466bfff640 1 -- 172.21.15.138:0/1107858514 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f467c01e350 con 0x7f468413dbc0 2024-09-11T05:08:12.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f466bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:08:12.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f466bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:08:12.577 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:08:12.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 10 librados: watch_flush enter 2024-09-11T05:08:12.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 10 librados: watch_flush exit 2024-09-11T05:08:12.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 20 client.4370.objecter shutdown clearing up homeless session... 2024-09-11T05:08:12.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 10 client.4370.objecter successfully canceled tick 2024-09-11T05:08:12.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 >> v2:172.21.15.190:6800/1562782189 conn(0x7f467003dcb0 msgr2=0x7f4670040170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 --2- 172.21.15.138:0/1107858514 >> v2:172.21.15.190:6800/1562782189 conn(0x7f467003dcb0 0x7f4670040170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4674001500 tx=0x7f4674007a40 comp rx=0 tx=0).stop 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 10 monclient: shutdown 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f468413dbc0 msgr2=0x7f468413e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 --2- 172.21.15.138:0/1107858514 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f468413dbc0 0x7f468413e040 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f468409dc90 tx=0x7f467c00be40 comp rx=0 tx=0).stop 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 shutdown_connections 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 --2- 172.21.15.138:0/1107858514 >> v2:172.21.15.190:6800/1562782189 conn(0x7f467003dcb0 0x7f4670040170 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 --2- 172.21.15.138:0/1107858514 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f468413dbc0 0x7f468413e040 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 --2- 172.21.15.138:0/1107858514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684138aa0 0x7f468413d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.574+0000 7f468a231640 1 --2- 172.21.15.138:0/1107858514 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f46700062d0 0x7f4684138560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.578+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 >> 172.21.15.138:0/1107858514 conn(0x7f46840aba10 msgr2=0x7f46840a5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.578+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 shutdown_connections 2024-09-11T05:08:12.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.578+0000 7f468a231640 1 -- 172.21.15.138:0/1107858514 wait complete. 2024-09-11T05:08:12.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.578+0000 7f468a231640 1 librados: shutdown 2024-09-11T05:08:12.595 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:08:12.596 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T05:08:12.596 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient: get_monmap_and_config 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient: build_initial_monmap 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient: monmap: 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:08:12.714013+0000 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:08:12.714013+0000 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.712 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:08:12.713 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:08:12.713 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:08:12.713 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:08:12.713 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:08:12.713 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 1 Processor -- start 2024-09-11T05:08:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 1 -- start start 2024-09-11T05:08:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient: init 2024-09-11T05:08:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-11T05:08:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f405c070010 0x7f405c135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.715 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient(hunting): picked mon.noname-a con 0x7f405c070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f405c09e6e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f405c09eb10 0x7f405c1389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient(hunting): picked mon.noname-c con 0x7f405c09eb10 addr v2:172.21.15.190:3300/0 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405c0a8990 con 0x7f405c09e6e0 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f405c0703f0 con 0x7f405c070010 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f405c09cbf0 con 0x7f405c09eb10 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f40638e7640 10 monclient(hunting): authenticate will time out at 976.150530s 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061683640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7f405c070010 msgr2=0x7f405c135450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061683640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f405c070010 0x7f405c135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:12.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f405c09eb10 0x7f405c1389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4060e82640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f405c09e6e0 0x7f405c1382c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.138:58496/0 (socket says 172.21.15.138:58496) 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f405c09eb10 0x7f405c1389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.138:38462/0 (socket says 172.21.15.138:38462) 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4060e82640 1 -- 172.21.15.138:0/3335888819 learned_addr learned my addr 172.21.15.138:0/3335888819 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient(hunting): get_auth_request con 0x7f405c09eb10 auth_method 0 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 1 -- 172.21.15.138:0/3335888819 >> v2:172.21.15.138:3300/0 conn(0x7f405c070010 msgr2=0x7f405c135450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 1 --2- 172.21.15.138:0/3335888819 >> v2:172.21.15.138:3300/0 conn(0x7f405c070010 0x7f405c135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 1 -- 172.21.15.138:0/3335888819 >> v1:172.21.15.190:6789/0 conn(0x7f405c09e6e0 legacy=0x7f405c1382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:08:12.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient: _finish_hunting 0 2024-09-11T05:08:12.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 1 monclient: found mon.noname-c 2024-09-11T05:08:12.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:12.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:08:12.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.710+0000 7f4061e84640 1 -- 172.21.15.138:0/3335888819 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405c09ced0 con 0x7f405c09eb10 2024-09-11T05:08:12.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 1 -- 172.21.15.138:0/3335888819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 msgr2=0x7f405c1389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 1 --2- 172.21.15.138:0/3335888819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 0x7f405c1389d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f405c135200 tx=0x7f405401c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff6c7a685802b3d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 1 -- 172.21.15.138:0/3335888819 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40540b2070 con 0x7f405c09eb10 2024-09-11T05:08:12.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: dump: 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: _finish_auth 0 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: _check_auth_tickets 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 5 monclient: authenticate success, global_id 4330 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 1 -- 172.21.15.138:0/3335888819 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40540a5960 con 0x7f405c09eb10 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient: get_monmap_and_config success 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 4 set_mon_vals no callback set 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 1 -- 172.21.15.138:0/3335888819 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40540a5d30 con 0x7f405c09eb10 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4053fff640 10 monclient: dump: 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.725 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient: shutdown 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- 172.21.15.138:0/3335888819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 msgr2=0x7f405c1389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 --2- 172.21.15.138:0/3335888819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 0x7f405c1389d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f405c135200 tx=0x7f405401c7e0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- 172.21.15.138:0/3335888819 shutdown_connections 2024-09-11T05:08:12.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 --2- 172.21.15.138:0/3335888819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 0x7f405c1389d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 --2- 172.21.15.138:0/3335888819 >> v2:172.21.15.138:3300/0 conn(0x7f405c070010 0x7f405c135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- 172.21.15.138:0/3335888819 >> 172.21.15.138:0/3335888819 conn(0x7f405c0ab830 msgr2=0x7f405c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- 172.21.15.138:0/3335888819 shutdown_connections 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- 172.21.15.138:0/3335888819 wait complete. 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient: build_initial_monmap 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient: monmap: 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:08:12.727 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 librados: starting msgr at 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 librados: starting objecter 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 Processor -- start 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- start start 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 librados: setting wanted keys 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 librados: calling monclient init 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient: init 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f405c070010 0x7f405c131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient(hunting): picked mon.noname-a con 0x7f405c070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:08:12.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09e6e0 0x7f405c132380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient(hunting): picked mon.noname-c con 0x7f405c09e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 0x7f405c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient(hunting): picked mon.noname-b con 0x7f405c09eb10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f405c142e80 con 0x7f405c070010 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f405c139120 con 0x7f405c09eb10 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f405c143000 con 0x7f405c09e6e0 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f40638e7640 10 monclient(hunting): authenticate will time out at 976.158803s 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061683640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f405c070010 msgr2=0x7f405c131e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061683640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f405c070010 0x7f405c131e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09e6e0 0x7f405c132380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09e6e0 0x7f405c132380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:39448/0 (socket says 172.21.15.138:39448) 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 -- 172.21.15.138:0/1236188952 learned_addr learned my addr 172.21.15.138:0/1236188952 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient(hunting): get_auth_request con 0x7f405c09e6e0 auth_method 0 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 0x7f405c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 10 monclient(hunting): get_auth_request con 0x7f405c09eb10 auth_method 0 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4061e84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 msgr2=0x7f405c0a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f405c09eb10 0x7f405c0a3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f405c070010 msgr2=0x7f405c131e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f405c070010 0x7f405c131e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient: _finish_hunting 0 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 monclient: found mon.noname-c 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:08:12.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.718+0000 7f4060e82640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405c133640 con 0x7f405c09e6e0 2024-09-11T05:08:12.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4060e82640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09e6e0 msgr2=0x7f405c132380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 client.?.objecter ms_handle_connect 0x7f405c09e6e0 2024-09-11T05:08:12.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:08:12.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4060e82640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09e6e0 0x7f405c132380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f404c00c470 tx=0x7f404c00c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c007070 con 0x7f405c09e6e0 2024-09-11T05:08:12.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:12.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: dump: 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: _finish_auth 0 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 5 monclient: authenticate success, global_id 4373 2024-09-11T05:08:12.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 10 monclient: _renew_subs 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404c007210 con 0x7f405c09e6e0 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f405c13b150 con 0x7f405c09e6e0 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c007600 con 0x7f405c09e6e0 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 monclient: dump: 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.738 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40537fe640 4 set_mon_vals no callback set 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 10 monclient: _renew_subs 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f405c13ae70 con 0x7f405c09e6e0 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 1 librados: init done 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 0x7f405c09f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f405c135450 con 0x7f405c09f0b0 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 0x7f405c09f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 10 monclient: get_auth_request con 0x7f405c09f0b0 auth_method 0 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061683640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f405c0008e0 con 0x7f405c09f0b0 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f404c01b400 con 0x7f405c09e6e0 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 1 --2- 172.21.15.138:0/1236188952 >> v2:172.21.15.190:6800/1562782189 conn(0x7f404003ceb0 0x7f404003f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f404c04ef30 con 0x7f405c09e6e0 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 10 client.4373.objecter ms_dispatch 0x7f405c000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 3 client.4373.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 3 client.4373.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4051ffb640 20 client.4373.objecter dump_active .. 0 homeless 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061e84640 1 --2- 172.21.15.138:0/1236188952 >> v2:172.21.15.190:6800/1562782189 conn(0x7f404003ceb0 0x7f404003f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.722+0000 7f4061e84640 10 monclient: get_auth_request con 0x7f404003ceb0 auth_method 0 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 msgr2=0x7f405c09f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 0x7f405c09f490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4058003990 tx=0x7f405800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 client.4373.objecter ms_handle_connect 0x7f405c09f0b0 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 client.4373.objecter resend_mon_ops 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 msgr2=0x7f405c09f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 msgr2=0x7f405c09f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 0x7f405c09f490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4058003990 tx=0x7f405800b8e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:12.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 0x7f405c09f490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4058003990 tx=0x7f405800b8e0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 msgr2=0x7f405c09f490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 0x7f405c09f490 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40400422a0 0x7f40400446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 msgr2=0x7f405c09f490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f0b0 0x7f405c09f490 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4040044d00 con 0x7f40400422a0 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40400422a0 0x7f40400446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 10 monclient: get_auth_request con 0x7f40400422a0 auth_method 0 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4040044eb0 con 0x7f40400422a0 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 client.4373.objecter ms_handle_connect 0x7f404003ceb0 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061e84640 1 --2- 172.21.15.138:0/1236188952 >> v2:172.21.15.190:6800/1562782189 conn(0x7f404003ceb0 0x7f404003f370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f40540af000 tx=0x7f40540a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 client.4373.objecter ms_handle_connect 0x7f40400422a0 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 client.4373.objecter resend_mon_ops 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40400422a0 msgr2=0x7f40400446c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4061683640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40400422a0 0x7f40400446c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4058009fd0 tx=0x7f40580059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058016070 con 0x7f40400422a0 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.726+0000 7f4051ffb640 10 monclient: dump: 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.730+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4058012dc0 con 0x7f40400422a0 2024-09-11T05:08:12.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.730+0000 7f4051ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:12.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.730+0000 7f4051ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:08:12.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.730+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40400422a0 msgr2=0x7f40400446c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.730+0000 7f4051ffb640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40400422a0 0x7f40400446c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4058009fd0 tx=0x7f40580059e0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.730+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40400422a0 msgr2=0x7f40400446c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:08:12.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.730+0000 7f4051ffb640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40400422a0 0x7f40400446c0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4058009fd0 tx=0x7f40580059e0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f40638e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:08:12.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f40638e7640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f9d0 0x7f405c0a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f40638e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f405c139950 con 0x7f405c09f9d0 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f9d0 0x7f405c0a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 10 monclient: get_auth_request con 0x7f405c09f9d0 auth_method 0 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.734+0000 7f4061683640 1 -- 172.21.15.138:0/1236188952 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f405c09fde0 con 0x7f405c09f9d0 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 10 client.4373.objecter ms_handle_connect 0x7f405c09f9d0 2024-09-11T05:08:12.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 10 client.4373.objecter resend_mon_ops 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4061683640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f9d0 msgr2=0x7f405c0a3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4061683640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f9d0 0x7f405c0a3c50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4040044e80 tx=0x7f405800f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058016040 con 0x7f405c09f9d0 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 10 monclient: dump: 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.748 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f4058032400 con 0x7f405c09f9d0 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f9d0 msgr2=0x7f405c0a3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f9d0 0x7f405c0a3c50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4040044e80 tx=0x7f405800f000 comp rx=0 tx=0).stop 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f9d0 msgr2=0x7f405c0a3c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4051ffb640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09f9d0 0x7f405c0a3c50 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4040044e80 tx=0x7f405800f000 comp rx=0 tx=0).stop 2024-09-11T05:08:12.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f4061e84640 1 -- 172.21.15.138:0/1236188952 reap_dead start 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 10, 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 51511, 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:08:12.750 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:08:12.751 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:08:12.752 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:08:12.753 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:08:12.754 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.756 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.757 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:08:12.758 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.759 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.760 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.762 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:08:12.762 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:08:12.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 10 librados: watch_flush enter 2024-09-11T05:08:12.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 10 librados: watch_flush exit 2024-09-11T05:08:12.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 20 client.4373.objecter shutdown clearing up homeless session... 2024-09-11T05:08:12.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 10 client.4373.objecter successfully canceled tick 2024-09-11T05:08:12.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 >> v2:172.21.15.190:6800/1562782189 conn(0x7f404003ceb0 msgr2=0x7f404003f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 --2- 172.21.15.138:0/1236188952 >> v2:172.21.15.190:6800/1562782189 conn(0x7f404003ceb0 0x7f404003f370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f40540af000 tx=0x7f40540a7070 comp rx=0 tx=0).stop 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 10 monclient: shutdown 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09e6e0 msgr2=0x7f405c132380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09e6e0 0x7f405c132380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f404c00c470 tx=0x7f404c00c930 comp rx=0 tx=0).stop 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 shutdown_connections 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 --2- 172.21.15.138:0/1236188952 >> v2:172.21.15.190:6800/1562782189 conn(0x7f404003ceb0 0x7f404003f370 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 --2- 172.21.15.138:0/1236188952 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f405c09e6e0 0x7f405c132380 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 >> 172.21.15.138:0/1236188952 conn(0x7f405c0ab830 msgr2=0x7f405c0a9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 shutdown_connections 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 -- 172.21.15.138:0/1236188952 wait complete. 2024-09-11T05:08:12.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.738+0000 7f40638e7640 1 librados: shutdown 2024-09-11T05:08:12.766 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:08:12.766 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T05:08:12.766 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:08:12.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient: get_monmap_and_config 2024-09-11T05:08:12.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient: build_initial_monmap 2024-09-11T05:08:12.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient: monmap: 2024-09-11T05:08:12.881 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:12.881 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.881 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:08:12.883151+0000 2024-09-11T05:08:12.881 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:08:12.883151+0000 2024-09-11T05:08:12.882 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:12.882 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.882 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:08:12.882 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:08:12.882 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:08:12.882 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:08:12.882 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:08:12.882 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 1 Processor -- start 2024-09-11T05:08:12.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 1 -- start start 2024-09-11T05:08:12.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient: init 2024-09-11T05:08:12.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:12.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T05:08:12.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa4fc0a32c0 0x7fa4fc09e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient(hunting): picked mon.noname-a con 0x7fa4fc0a32c0 addr v2:172.21.15.138:3300/0 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4fc09e960 addr v1:172.21.15.190:6789/0 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4fc09ed70 0x7fa4fc13ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient(hunting): picked mon.noname-c con 0x7fa4fc09ed70 addr v2:172.21.15.190:3300/0 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4fc0a1920 con 0x7fa4fc09e960 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fa4fc0744a0 con 0x7fa4fc0a32c0 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa4fc0a36a0 con 0x7fa4fc09ed70 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa502778640 10 monclient(hunting): authenticate will time out at 976.319160s 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa4fbfff640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7fa4fc0a32c0 msgr2=0x7fa4fc09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa4fbfff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa4fc0a32c0 0x7fa4fc09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:12.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa500d15640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa4fc09ed70 0x7fa4fc13ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa4fb7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa4fc09e960 0x7fa4fc139710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.138:58500/0 (socket says 172.21.15.138:58500) 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa4fb7fe640 1 -- 172.21.15.138:0/68009385 learned_addr learned my addr 172.21.15.138:0/68009385 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa500d15640 10 monclient(hunting): get_auth_request con 0x7fa4fc09ed70 auth_method 0 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa500d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa500d15640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa500d15640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa500d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa500d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.878+0000 7fa500d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 1 -- 172.21.15.138:0/68009385 >> v2:172.21.15.138:3300/0 conn(0x7fa4fc0a32c0 msgr2=0x7fa4fc09e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 1 --2- 172.21.15.138:0/68009385 >> v2:172.21.15.138:3300/0 conn(0x7fa4fc0a32c0 0x7fa4fc09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 1 -- 172.21.15.138:0/68009385 >> v1:172.21.15.190:6789/0 conn(0x7fa4fc09e960 legacy=0x7fa4fc139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 10 monclient: _finish_hunting 0 2024-09-11T05:08:12.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 1 monclient: found mon.noname-c 2024-09-11T05:08:12.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:12.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:08:12.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.882+0000 7fa500d15640 1 -- 172.21.15.138:0/68009385 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4fc0644b0 con 0x7fa4fc09ed70 2024-09-11T05:08:12.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 -- 172.21.15.138:0/68009385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 msgr2=0x7fa4fc13ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 --2- 172.21.15.138:0/68009385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 0x7fa4fc13ce40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc09f7b0 tx=0x7fa4f401c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dabf7fc639c0f3c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 1 -- 172.21.15.138:0/68009385 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f40ac070 con 0x7fa4fc09ed70 2024-09-11T05:08:12.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:12.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: dump: 2024-09-11T05:08:12.892 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.892 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.892 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.892 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: _finish_auth 0 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: _check_auth_tickets 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 1 -- 172.21.15.138:0/68009385 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f40a6720 con 0x7fa4fc09ed70 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 1 -- 172.21.15.138:0/68009385 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f40a6af0 con 0x7fa4fc09ed70 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4faffd640 10 monclient: dump: 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 5 monclient: authenticate success, global_id 4336 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient: get_monmap_and_config success 2024-09-11T05:08:12.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 4 set_mon_vals no callback set 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient: shutdown 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- 172.21.15.138:0/68009385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 msgr2=0x7fa4fc13ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 --2- 172.21.15.138:0/68009385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 0x7fa4fc13ce40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc09f7b0 tx=0x7fa4f401c6c0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- 172.21.15.138:0/68009385 shutdown_connections 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 --2- 172.21.15.138:0/68009385 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 0x7fa4fc13ce40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 --2- 172.21.15.138:0/68009385 >> v2:172.21.15.138:3300/0 conn(0x7fa4fc0a32c0 0x7fa4fc09e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- 172.21.15.138:0/68009385 >> 172.21.15.138:0/68009385 conn(0x7fa4fc063640 msgr2=0x7fa4fc063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- 172.21.15.138:0/68009385 shutdown_connections 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- 172.21.15.138:0/68009385 wait complete. 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient: build_initial_monmap 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient: monmap: 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:08:12.895 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 librados: starting msgr at 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 librados: starting objecter 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 Processor -- start 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- start start 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 librados: setting wanted keys 2024-09-11T05:08:12.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 librados: calling monclient init 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient: init 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient: _reopen_session rank -1 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4fc09e960 0x7fa4fc1cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient(hunting): picked mon.noname-a con 0x7fa4fc09e960 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 0x7fa4fc121040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient(hunting): picked mon.noname-b con 0x7fa4fc09ed70 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 0x7fa4fc138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient(hunting): picked mon.noname-c con 0x7fa4fc0a32c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc0a21b0 con 0x7fa4fc09e960 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc09f290 con 0x7fa4fc09ed70 2024-09-11T05:08:12.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa4fc0a2330 con 0x7fa4fc0a32c0 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient(hunting): _renew_subs 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa502778640 10 monclient(hunting): authenticate will time out at 976.327097s 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fbfff640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4fc09e960 msgr2=0x7fa4fc1cc540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fbfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4fc09e960 0x7fa4fc1cc540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 0x7fa4fc138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 0x7fa4fc138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:39494/0 (socket says 172.21.15.138:39494) 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 -- 172.21.15.138:0/949921944 learned_addr learned my addr 172.21.15.138:0/949921944 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 10 monclient(hunting): get_auth_request con 0x7fa4fc0a32c0 auth_method 0 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 0x7fa4fc121040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 10 monclient(hunting): get_auth_request con 0x7fa4fc09ed70 auth_method 0 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 msgr2=0x7fa4fc138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 msgr2=0x7fa4fc138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 0x7fa4fc138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa500d15640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 0x7fa4fc138dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.886+0000 7fa4fb7fe640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 msgr2=0x7fa4fc138dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa4fc0a32c0 0x7fa4fc138dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4fc09e960 msgr2=0x7fa4fc1cc540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa4fc09e960 0x7fa4fc1cc540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 1 monclient: found mon.noname-b 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.190:3300/0 2024-09-11T05:08:12.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 1 -- 172.21.15.138:0/949921944 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4fc1219d0 con 0x7fa4fc09ed70 2024-09-11T05:08:12.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 msgr2=0x7fa4fc121040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fb7fe640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 0x7fa4fc121040 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00b440 tx=0x7fa4ec00b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa4fc09ed70 2024-09-11T05:08:12.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:08:12.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec015020 con 0x7fa4fc09ed70 2024-09-11T05:08:12.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:08:12.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: dump: 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: _finish_auth 0 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:12.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 5 monclient: authenticate success, global_id 4339 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 10 monclient: _renew_subs 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4fc122840 con 0x7fa4fc09ed70 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4ec011a60 con 0x7fa4fc09ed70 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec011e50 con 0x7fa4fc09ed70 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4f8ff9640 10 monclient: dump: 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.907 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 10 monclient: _renew_subs 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fa7fc640 4 set_mon_vals no callback set 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.190:3300/0 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4fc11ffb0 con 0x7fa4fc09ed70 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 1 librados: init done 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc139300 0x7fa4fc1c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc09e420 con 0x7fa4fc139300 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fbfff640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc139300 0x7fa4fc1c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fbfff640 10 monclient: get_auth_request con 0x7fa4fc139300 auth_method 0 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.890+0000 7fa4fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fa4ec01a400 con 0x7fa4fc09ed70 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4fbfff640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-09-11T05:08:12.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4fbfff640 1 -- 172.21.15.138:0/949921944 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4fc0008e0 con 0x7fa4fc139300 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 --2- 172.21.15.138:0/949921944 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa4e803cff0 0x7fa4e803f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7fa4ec04f460 con 0x7fa4fc09ed70 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 client.4339.objecter ms_dispatch 0x7fa4fc000ba0 osd_map(20..20 src has 1..20) 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 3 client.4339.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 3 client.4339.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 20 client.4339.objecter dump_active .. 0 homeless 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa500d15640 1 --2- 172.21.15.138:0/949921944 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa4e803cff0 0x7fa4e803f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa500d15640 10 monclient: get_auth_request con 0x7fa4e803cff0 auth_method 0 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 client.4339.objecter ms_handle_connect 0x7fa4fc139300 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4fbfff640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc139300 msgr2=0x7fa4fc1c9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 client.4339.objecter resend_mon_ops 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4fbfff640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc139300 0x7fa4fc1c9290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000b450 tx=0x7fa4f000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0005270 con 0x7fa4fc139300 2024-09-11T05:08:12.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 monclient: dump: 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa4f0015070 con 0x7fa4fc139300 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc139300 msgr2=0x7fa4fc1c9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc139300 0x7fa4fc1c9290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000b450 tx=0x7fa4f000b910 comp rx=0 tx=0).stop 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc139300 msgr2=0x7fa4fc1c9290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc139300 0x7fa4fc1c9290 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa500d15640 1 --2- 172.21.15.138:0/949921944 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa4e803cff0 0x7fa4e803f4b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc06f810 tx=0x7fa4f40a7000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.894+0000 7fa4f8ff9640 10 client.4339.objecter ms_handle_connect 0x7fa4e803cff0 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa502778640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa502778640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc02d450 0x7fa4fc02d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa502778640 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc13b0e0 con 0x7fa4fc02d450 2024-09-11T05:08:12.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc02d450 0x7fa4fc02d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 10 monclient: get_auth_request con 0x7fa4fc02d450 auth_method 0 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-09-11T05:08:12.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.902+0000 7fa4fbfff640 1 -- 172.21.15.138:0/949921944 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4fc0a6270 con 0x7fa4fc02d450 2024-09-11T05:08:12.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4fbfff640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc02d450 msgr2=0x7fa4fc02d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:12.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4fbfff640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc02d450 0x7fa4fc02d860 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000b450 tx=0x7fa4f00059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:12.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 10 client.4339.objecter ms_handle_connect 0x7fa4fc02d450 2024-09-11T05:08:12.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 10 client.4339.objecter resend_mon_ops 2024-09-11T05:08:12.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0010690 con 0x7fa4fc02d450 2024-09-11T05:08:12.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 10 monclient: dump: 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7fa4f0010690 con 0x7fa4fc02d450 2024-09-11T05:08:12.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc02d450 msgr2=0x7fa4fc02d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc02d450 0x7fa4fc02d860 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000b450 tx=0x7fa4f00059e0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc02d450 msgr2=0x7fa4fc02d860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa4f8ff9640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc02d450 0x7fa4fc02d860 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:08:12.916 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 10, 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 51643, 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:08:12.917 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:08:12.918 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:08:12.919 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:08:12.920 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:08:12.921 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.922 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:08:12.923 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:08:12.925 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.926 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:08:12.927 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:08:12.928 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:08:12.929 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:08:12.929 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:08:12.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:08:12.929 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:08:12.929 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 10 librados: watch_flush enter 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 10 librados: watch_flush exit 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 20 client.4339.objecter shutdown clearing up homeless session... 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 10 client.4339.objecter successfully canceled tick 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa4e803cff0 msgr2=0x7fa4e803f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 --2- 172.21.15.138:0/949921944 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa4e803cff0 0x7fa4e803f4b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc06f810 tx=0x7fa4f40a7000 comp rx=0 tx=0).stop 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 10 monclient: shutdown 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 msgr2=0x7fa4fc121040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 --2- 172.21.15.138:0/949921944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4fc09ed70 0x7fa4fc121040 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00b440 tx=0x7fa4ec00b900 comp rx=0 tx=0).stop 2024-09-11T05:08:12.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa500d15640 1 -- 172.21.15.138:0/949921944 reap_dead start 2024-09-11T05:08:12.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa500d15640 1 -- 172.21.15.138:0/949921944 reap_dead start 2024-09-11T05:08:12.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 shutdown_connections 2024-09-11T05:08:12.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 >> 172.21.15.138:0/949921944 conn(0x7fa4fc063640 msgr2=0x7fa4fc0a5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:12.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 shutdown_connections 2024-09-11T05:08:12.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 -- 172.21.15.138:0/949921944 wait complete. 2024-09-11T05:08:12.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:08:12.906+0000 7fa502778640 1 librados: shutdown 2024-09-11T05:08:12.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:08:12.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:08:44.430 INFO:tasks.workunit.client.0.smithi190.stderr:Note: switching to '8f219c476cb4f26fc5719c661c0323719a45cb30'. 2024-09-11T05:08:44.430 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T05:08:44.430 INFO:tasks.workunit.client.0.smithi190.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T05:08:44.430 INFO:tasks.workunit.client.0.smithi190.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T05:08:44.430 INFO:tasks.workunit.client.0.smithi190.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr: git switch -c 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr:Or undo this operation with: 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr: git switch - 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T05:08:44.431 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T05:08:44.432 INFO:tasks.workunit.client.0.smithi190.stderr:HEAD is now at 8f219c476cb Merge branch 'mgr-vol-mod-disable' of https://github.com/rishabh-d-dave/ceph into wip-hemanth-testing-2024-09-10-0723 2024-09-11T05:08:44.435 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T05:08:44.485 INFO:tasks.workunit.client.0.smithi190.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T05:08:44.488 INFO:tasks.workunit.client.0.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T05:08:44.489 INFO:tasks.workunit.client.0.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T05:08:45.273 INFO:tasks.workunit.client.0.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T05:08:45.349 INFO:tasks.workunit.client.0.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T05:08:45.406 INFO:tasks.workunit.client.0.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T05:08:45.407 INFO:tasks.workunit.client.0.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T05:08:45.407 INFO:tasks.workunit.client.0.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T05:08:45.449 INFO:tasks.workunit.client.0.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T05:08:45.452 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T05:08:45.452 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T05:08:45.499 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-11T05:08:45.500 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-11T05:08:45.500 DEBUG:teuthology.orchestra.run.smithi190:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8f219c476cb4f26fc5719c661c0323719a45cb30 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ parallel=1 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' '' = --serial ']' 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ crimson=0 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' '' = --crimson ']' 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ color= 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -t 1 ']' 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-11T05:08:45.553 INFO:tasks.workunit.client.0.smithi190.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-11T05:08:45.555 INFO:tasks.workunit.client.0.smithi190.stderr:+ declare -A pids 2024-09-11T05:08:45.555 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.555 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.555 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_aio 2024-09-11T05:08:45.555 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_aio' 2024-09-11T05:08:45.556 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_aio 2024-09-11T05:08:45.556 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.558 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_aio 2024-09-11T05:08:45.558 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28036 2024-09-11T05:08:45.558 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_aio on pid 28036' 2024-09-11T05:08:45.558 INFO:tasks.workunit.client.0.smithi190.stdout:test api_aio on pid 28036 2024-09-11T05:08:45.559 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28036 2024-09-11T05:08:45.559 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.559 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.559 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-11T05:08:45.559 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_aio_pp 2024-09-11T05:08:45.559 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_aio_pp' 2024-09-11T05:08:45.560 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_aio_pp 2024-09-11T05:08:45.560 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.560 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.560 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.560 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.560 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.561 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-11T05:08:45.561 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-11T05:08:45.561 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_aio: /' 2024-09-11T05:08:45.561 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_aio_pp 2024-09-11T05:08:45.561 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28044 2024-09-11T05:08:45.561 INFO:tasks.workunit.client.0.smithi190.stdout:test api_aio_pp on pid 28044 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_aio_pp on pid 28044' 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28044 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_io 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_io' 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.563 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.564 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.564 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_io 2024-09-11T05:08:45.564 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.564 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-11T05:08:45.564 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-11T05:08:45.564 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-11T05:08:45.564 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_io 2024-09-11T05:08:45.565 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28052 2024-09-11T05:08:45.568 INFO:tasks.workunit.client.0.smithi190.stdout:test api_io on pid 28052 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_io on pid 28052' 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28052 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_io_pp 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_io_pp' 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_io_pp 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.570 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_io.log 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_io: /' 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_io_pp 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28060 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_io_pp on pid 28060' 2024-09-11T05:08:45.571 INFO:tasks.workunit.client.0.smithi190.stdout:test api_io_pp on pid 28060 2024-09-11T05:08:45.573 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28060 2024-09-11T05:08:45.573 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.573 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.573 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-11T05:08:45.573 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_asio 2024-09-11T05:08:45.573 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_asio' 2024-09-11T05:08:45.573 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.573 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_asio 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_asio 2024-09-11T05:08:45.574 INFO:tasks.workunit.client.0.smithi190.stdout:test api_asio on pid 28068 2024-09-11T05:08:45.576 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28068 2024-09-11T05:08:45.576 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_asio on pid 28068' 2024-09-11T05:08:45.576 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28068 2024-09-11T05:08:45.576 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.576 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.576 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-11T05:08:45.576 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_list 2024-09-11T05:08:45.576 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_list' 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_list 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_asio: /' 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_list 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28076 2024-09-11T05:08:45.577 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_list on pid 28076' 2024-09-11T05:08:45.578 INFO:tasks.workunit.client.0.smithi190.stdout:test api_list on pid 28076 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28076 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_lock 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_lock' 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_lock 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.579 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.580 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.580 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-11T05:08:45.580 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_list.log 2024-09-11T05:08:45.583 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.583 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_list: /' 2024-09-11T05:08:45.585 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_lock 2024-09-11T05:08:45.585 INFO:tasks.workunit.client.0.smithi190.stdout:test api_lock on pid 28084 2024-09-11T05:08:45.586 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28084 2024-09-11T05:08:45.586 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_lock on pid 28084' 2024-09-11T05:08:45.586 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28084 2024-09-11T05:08:45.586 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.586 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.586 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-11T05:08:45.586 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_lock_pp 2024-09-11T05:08:45.587 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_lock_pp' 2024-09-11T05:08:45.587 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.587 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.587 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.587 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_lock_pp 2024-09-11T05:08:45.587 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.588 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.588 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-11T05:08:45.589 INFO:tasks.workunit.client.0.smithi190.stdout:test api_lock_pp on pid 28092 2024-09-11T05:08:45.590 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_lock_pp 2024-09-11T05:08:45.590 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28092 2024-09-11T05:08:45.590 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_lock_pp on pid 28092' 2024-09-11T05:08:45.590 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28092 2024-09-11T05:08:45.590 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.590 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.590 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-11T05:08:45.591 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.591 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.591 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.592 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.592 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-11T05:08:45.596 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-11T05:08:45.596 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-11T05:08:45.597 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_misc 2024-09-11T05:08:45.597 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_misc' 2024-09-11T05:08:45.601 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.602 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-11T05:08:45.603 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_lock: /' 2024-09-11T05:08:45.603 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_misc 2024-09-11T05:08:45.604 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_misc 2024-09-11T05:08:45.604 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28114 2024-09-11T05:08:45.605 INFO:tasks.workunit.client.0.smithi190.stdout:test api_misc on pid 28114 2024-09-11T05:08:45.605 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_misc on pid 28114' 2024-09-11T05:08:45.605 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28114 2024-09-11T05:08:45.605 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.605 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.606 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_misc_pp 2024-09-11T05:08:45.606 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_misc_pp' 2024-09-11T05:08:45.609 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_misc_pp 2024-09-11T05:08:45.610 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.610 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-11T05:08:45.612 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.612 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.612 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.612 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.612 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-11T05:08:45.612 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_misc: /' 2024-09-11T05:08:45.613 INFO:tasks.workunit.client.0.smithi190.stdout:test api_misc_pp on pid 28149 2024-09-11T05:08:45.613 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_misc_pp 2024-09-11T05:08:45.613 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-11T05:08:45.613 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28149 2024-09-11T05:08:45.613 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_misc_pp on pid 28149' 2024-09-11T05:08:45.614 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28149 2024-09-11T05:08:45.614 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.614 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.614 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-11T05:08:45.614 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_tier_pp 2024-09-11T05:08:45.614 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_tier_pp' 2024-09-11T05:08:45.614 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_tier_pp 2024-09-11T05:08:45.616 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.616 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.616 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.616 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.617 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-11T05:08:45.618 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-11T05:08:45.618 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.619 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-11T05:08:45.621 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_tier_pp 2024-09-11T05:08:45.621 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28180 2024-09-11T05:08:45.621 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_tier_pp on pid 28180' 2024-09-11T05:08:45.621 INFO:tasks.workunit.client.0.smithi190.stdout:test api_tier_pp on pid 28180 2024-09-11T05:08:45.622 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28180 2024-09-11T05:08:45.622 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.622 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.622 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-11T05:08:45.622 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_pool 2024-09-11T05:08:45.622 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_pool' 2024-09-11T05:08:45.623 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.623 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.623 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.623 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.623 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-11T05:08:45.623 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-11T05:08:45.637 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-11T05:08:45.637 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [----------] Global test environment set-up. 2024-09-11T05:08:45.637 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-11T05:08:45.637 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient: get_monmap_and_config 2024-09-11T05:08:45.637 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient: build_initial_monmap 2024-09-11T05:08:45.637 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient: monmap: 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 0 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 0.000000 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 0.000000 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 1 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 1 Processor -- start 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 1 -- start start 2024-09-11T05:08:45.638 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient: init 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient: _reopen_session rank -1 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cbaa8f0 0x555a8cba9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient(hunting): picked mon.noname-c con 0x555a8cbaa8f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cba9540 0x555a8cba9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient(hunting): picked mon.noname-a con 0x555a8cba9540 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cc3feb0 0x555a8cc422a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient(hunting): picked mon.noname-b con 0x555a8cc3feb0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555a8caa2f30 con 0x555a8cba9540 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555a8ca9f120 con 0x555a8cc3feb0 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:45.639 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x555a8ca449f0 con 0x555a8cbaa8f0 2024-09-11T05:08:45.640 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient(hunting): _renew_subs 2024-09-11T05:08:45.640 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d70de6880 10 monclient(hunting): authenticate will time out at 1006.430102s 2024-09-11T05:08:45.640 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-11T05:08:45.640 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e0ef640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cba9540 msgr2=0x555a8cba9920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:45.640 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e0ef640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cba9540 0x555a8cba9920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cbaa8f0 0x555a8cba9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cbaa8f0 0x555a8cba9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.190:60008/0 (socket says 172.21.15.190:60008) 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/3854608637 learned_addr learned my addr 172.21.15.190:0/3854608637 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient(hunting): get_auth_request con 0x555a8cbaa8f0 auth_method 0 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:45.641 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6f0f1640 1 --2- 172.21.15.190:0/3854608637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cc3feb0 0x555a8cc422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6f0f1640 10 monclient(hunting): get_auth_request con 0x555a8cc3feb0 auth_method 0 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6f0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6f0f1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6f0f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient(hunting): handle_auth_done global_id 4403 payload 274 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/3854608637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cc3feb0 msgr2=0x555a8cc422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 --2- 172.21.15.190:0/3854608637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cc3feb0 0x555a8cc422a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:45.642 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/3854608637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cba9540 msgr2=0x555a8cba9920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:08:45.643 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_pool 2024-09-11T05:08:45.643 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.643 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 --2- 172.21.15.190:0/3854608637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cba9540 0x555a8cba9920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:45.643 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient: _finish_hunting 0 2024-09-11T05:08:45.643 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 monclient: found mon.noname-c 2024-09-11T05:08:45.643 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:45.643 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:08:45.643 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.634+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/3854608637 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a8ca9f2a0 con 0x555a8cbaa8f0 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/3854608637 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cbaa8f0 msgr2=0x555a8cba9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6e8f0640 1 --2- 172.21.15.190:0/3854608637 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cbaa8f0 0x555a8cba9000 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d6001ea70 tx=0x7f6d60002120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e85b99620a0be80 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 1 -- 172.21.15.190:0/3854608637 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600a8d20 con 0x555a8cbaa8f0 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 10 monclient: dump: 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 1 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 19 (squid) 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 1 2024-09-11T05:08:45.644 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 10 monclient: _finish_auth 0 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 10 monclient: _check_auth_tickets 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 5 monclient: authenticate success, global_id 4403 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 1 -- 172.21.15.190:0/3854608637 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d60001af0 con 0x555a8cbaa8f0 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d6d0ed640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 10 monclient: get_monmap_and_config success 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 4 set_mon_vals no callback set 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 10 monclient: shutdown 2024-09-11T05:08:45.645 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 20 monclient: shutdotest api_pool on pid 28224 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_pool 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28224 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_pool on pid 28224' 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28224 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_snapshots 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_snapshots' 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_snapshots 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.647 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stdout:wn discarding 0 pending message(s) 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 1 -- 172.21.15.190:0/3854608637 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cbaa8f0 msgr2=0x555a8cba9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 1 --2- 172.21.15.190:0/3854608637 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cbaa8f0 0x555a8cba9000 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d6001ea70 tx=0x7f6d60002120 comp rx=0 tx=0).stop 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 1 -- 172.21.15.190:0/3854608637 shutdown_connections 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 1 --2- 172.21.15.190:0/3854608637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cc3feb0 0x555a8cc422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 1 --2- 172.21.15.190:0/3854608637 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cba9540 0x555a8cba9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 1 --2- 172.21.15.190:0/3854608637 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cbaa8f0 0x555a8cba9000 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:45.648 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.638+0000 7f6d70de6880 1 -- 172.21.15.190:0/3854608637 >> 172.21.15.190:0/3854608637 conn(0x555a8caf8480 msgr2=0x555a8cba3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 -- 172.21.15.190:0/3854608637 shutdown_connections 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 -- 172.21.15.190:0/3854608637 wait complete. 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient: build_initial_monmap 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient: monmap: 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 0 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 0.000000 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 0.000000 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 1 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:08:45.649 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 librados: starting msgr at 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 librados: starting objecter 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 Processor -- start 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 -- start start 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 librados: setting wanted keys 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 librados: calling monclient init 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient: init 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient: _reopen_session rank -1 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:08:45.650 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cba9540 0x555a8cc3f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:45.651 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-11T05:08:45.651 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.651 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient(hunting): picked mon.noname-c con 0x555a8cba9540 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:08:45.651 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cbaa8f0 0x555a8cc3f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:45.651 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient(hunting): picked mon.noname-a con 0x555a8cbaa8f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:08:45.651 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cc3feb0 0x555a8cba33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient(hunting): picked mon.noname-b con 0x555a8cc3feb0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x555a8cba9e60 con 0x555a8cbaa8f0 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x555a8cb18030 con 0x555a8cc3feb0 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient(hunting): start opening mon connection 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x555a8cb17e30 con 0x555a8cba9540 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient(hunting): _renew_subs 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.642+0000 7f6d70de6880 10 monclient(hunting): authenticate will time out at 1006.439840s 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e0ef640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cbaa8f0 msgr2=0x555a8cc3f930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e0ef640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cbaa8f0 0x555a8cc3f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cba9540 0x555a8cc3f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cba9540 0x555a8cc3f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.190:60022/0 (socket says 172.21.15.190:60022) 2024-09-11T05:08:45.652 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/2305799564 learned_addr learned my addr 172.21.15.190:0/2305799564 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T05:08:45.653 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient(hunting): get_auth_request con 0x555a8cba9540 auth_method 0 2024-09-11T05:08:45.653 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:08:45.653 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:08:45.653 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:08:45.653 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:08:45.653 INFO:tasks.workunit.client.0.smithi190.stdout: test api_snapshots on pid 28242 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_snapshots 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28242 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_snapshots on pid 28242' 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28242 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.654 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/2305799564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cc3feb0 msgr2=0x555a8cba33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 --2- 172.21.15.190:0/2305799564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555a8cc3feb0 0x555a8cba33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/2305799564 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cbaa8f0 msgr2=0x555a8cc3f930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 --2- 172.21.15.190:0/2305799564 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x555a8cbaa8f0 0x555a8cc3f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:45.655 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient: _finish_hunting 0 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 monclient: found mon.noname-c 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/2305799564 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a8cb0cb20 con 0x555a8cba9540 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 -- 172.21.15.190:0/2305799564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cba9540 msgr2=0x555a8cc3f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6e8f0640 1 --2- 172.21.15.190:0/2305799564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cba9540 0x555a8cc3f3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d60001470 tx=0x7f6d5c004130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 client.?.objecter ms_handle_connect 0x555a8cba9540 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600c3070 con 0x555a8cba9540 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: dump: 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 1 2024-09-11T05:08:45.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:45.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:45.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:45.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 19 (squid) 2024-09-11T05:08:45.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 1 2024-09-11T05:08:45.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:45.657 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-11T05:08:45.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: _finish_auth 0 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: _check_auth_tickets 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d60001af0 con 0x555a8cba9540 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600b0b00 con 0x555a8cba9540 2024-09-11T05:08:45.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d57fff640 10 monclient: dump: 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 1 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 19 (squid) 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 1 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:08:45.659 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.646+0000 7f6d6d0ed640 4 set_mon_vals no callback set 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 5 monclient: authenticate success, global_id 4409 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 10 monclient: _renew_subs 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x555a8cb0cdc0 con 0x555a8cba9540 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 10 monclient: _renew_subs 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x555a8cb17460 con 0x555a8cba9540 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 1 librados: init done 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 10 librados: wait_for_osdmap waiting 2024-09-11T05:08:45.660 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f6d600b0ce0 con 0x555a8cba9540 2024-09-11T05:08:45.661 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d57fff640 1 --2- 172.21.15.190:0/2305799564 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d5003b0f0 0x7f6d5003d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:45.661 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_pool: /' 2024-09-11T05:08:45.661 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_snapshots_pp 2024-09-11T05:08:45.661 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_snapshots_pp' 2024-09-11T05:08:45.661 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_snapshots_pp 2024-09-11T05:08:45.662 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.662 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(20..20 test api_snapshots_pp on pid 28276 2024-09-11T05:08:45.662 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_snapshots_pp 2024-09-11T05:08:45.662 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28276 2024-09-11T05:08:45.662 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_snapshots_pp on pid 28276' 2024-09-11T05:08:45.663 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28276 2024-09-11T05:08:45.663 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.663 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.663 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_stat 2024-09-11T05:08:45.663 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_stat' 2024-09-11T05:08:45.663 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-11T05:08:45.665 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.665 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.665 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.665 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.665 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-11T05:08:45.665 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-11T05:08:45.666 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-11T05:08:45.669 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_stat 2024-09-11T05:08:45.670 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.672 INFO:tasks.workunit.client.0.smithi190.stdout:test api_stat on pid 28297 2024-09-11T05:08:45.673 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_stat 2024-09-11T05:08:45.673 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28297 2024-09-11T05:08:45.673 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_stat on pid 28297' 2024-09-11T05:08:45.673 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28297 2024-09-11T05:08:45.673 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.673 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.673 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_stat_pp 2024-09-11T05:08:45.673 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_stat_pp' 2024-09-11T05:08:45.674 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-11T05:08:45.674 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.674 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.675 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.675 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.675 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.675 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-11T05:08:45.675 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_stat_pp 2024-09-11T05:08:45.676 INFO:tasks.workunit.client.0.smithi190.stdout:test api_stat_pp on pid 28306 2024-09-11T05:08:45.677 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_stat_pp 2024-09-11T05:08:45.677 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28306 2024-09-11T05:08:45.677 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_stat_pp on pid 28306' 2024-09-11T05:08:45.677 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28306 2024-09-11T05:08:45.677 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.677 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.677 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_watch_notify 2024-09-11T05:08:45.677 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_watch_notify' 2024-09-11T05:08:45.678 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-11T05:08:45.678 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.679 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.679 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.679 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.679 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-11T05:08:45.680 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.682 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-11T05:08:45.682 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_stat: /' 2024-09-11T05:08:45.682 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-11T05:08:45.683 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-11T05:08:45.684 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_watch_notify 2024-09-11T05:08:45.685 INFO:tasks.workunit.client.0.smithi190.stdout:test api_watch_notify on pid 28323 2024-09-11T05:08:45.686 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_watch_notify 2024-09-11T05:08:45.686 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28323 2024-09-11T05:08:45.686 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_watch_notify on pid 28323' 2024-09-11T05:08:45.686 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28323 2024-09-11T05:08:45.686 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.686 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.686 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-11T05:08:45.687 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.687 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.687 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.687 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.688 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_watch_notify_pp 2024-09-11T05:08:45.688 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_watch_notify_pp' 2024-09-11T05:08:45.689 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-11T05:08:45.689 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-11T05:08:45.689 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-11T05:08:45.692 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_watch_notify_pp 2024-09-11T05:08:45.693 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.696 INFO:tasks.workunit.client.0.smithi190.stdout:test api_watch_notify_pp on pid 28353 2024-09-11T05:08:45.696 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_watch_notify_pp 2024-09-11T05:08:45.696 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28353 2024-09-11T05:08:45.696 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_watch_notify_pp on pid 28353' 2024-09-11T05:08:45.696 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28353 2024-09-11T05:08:45.696 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.696 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.696 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-11T05:08:45.697 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.697 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.697 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.697 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.698 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-11T05:08:45.701 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_cmd 2024-09-11T05:08:45.701 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_cmd' 2024-09-11T05:08:45.704 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-11T05:08:45.704 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-11T05:08:45.710 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_cmd 2024-09-11T05:08:45.717 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.720 INFO:tasks.workunit.client.0.smithi190.stdout:test api_cmd on pid 28428 2024-09-11T05:08:45.720 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_cmd 2024-09-11T05:08:45.720 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28428 2024-09-11T05:08:45.720 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_cmd on pid 28428' 2024-09-11T05:08:45.720 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28428 2024-09-11T05:08:45.720 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.721 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.721 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_cmd_pp 2024-09-11T05:08:45.721 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_cmd_pp' 2024-09-11T05:08:45.721 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-11T05:08:45.721 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.723 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.723 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.723 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.723 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.723 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_cmd_pp 2024-09-11T05:08:45.724 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-11T05:08:45.724 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_cmd_pp 2024-09-11T05:08:45.724 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-11T05:08:45.724 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28442 2024-09-11T05:08:45.724 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_cmd_pp on pid 28442' 2024-09-11T05:08:45.725 INFO:tasks.workunit.client.0.smithi190.stdout:test api_cmd_pp on pid 28442 2024-09-11T05:08:45.725 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28442 2024-09-11T05:08:45.725 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.725 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.725 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-11T05:08:45.726 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.726 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.727 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.727 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.727 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-11T05:08:45.727 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-11T05:08:45.727 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_service 2024-09-11T05:08:45.727 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_service' 2024-09-11T05:08:45.731 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-11T05:08:45.733 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_service 2024-09-11T05:08:45.733 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.734 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_cmd: /' 2024-09-11T05:08:45.735 INFO:tasks.workunit.client.0.smithi190.stdout:test api_service on pid 28469 2024-09-11T05:08:45.736 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_service 2024-09-11T05:08:45.736 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28469 2024-09-11T05:08:45.736 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_service on pid 28469' 2024-09-11T05:08:45.736 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28469 2024-09-11T05:08:45.736 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.736 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.737 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_service_pp 2024-09-11T05:08:45.737 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-11T05:08:45.737 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_service_pp' 2024-09-11T05:08:45.737 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.738 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.738 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.738 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.738 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-11T05:08:45.738 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_service.log 2024-09-11T05:08:45.741 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_service: /' 2024-09-11T05:08:45.742 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_service_pp 2024-09-11T05:08:45.742 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.746 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_service_pp 2024-09-11T05:08:45.746 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28502 2024-09-11T05:08:45.746 INFO:tasks.workunit.client.0.smithi190.stdout:test api_service_pp on pid 28502 2024-09-11T05:08:45.747 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_service_pp on pid 28502' 2024-09-11T05:08:45.747 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28502 2024-09-11T05:08:45.747 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.747 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.747 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_c_write_operations 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_c_write_operations' 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_c_write_operations 2024-09-11T05:08:45.748 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-11T05:08:45.749 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-11T05:08:45.750 INFO:tasks.workunit.client.0.smithi190.stdout:test api_c_write_operations on pid 28511 2024-09-11T05:08:45.750 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_c_write_operations 2024-09-11T05:08:45.750 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28511 2024-09-11T05:08:45.750 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_c_write_operations on pid 28511' 2024-09-11T05:08:45.751 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28511 2024-09-11T05:08:45.751 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.751 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.751 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-11T05:08:45.751 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_c_read_operations 2024-09-11T05:08:45.751 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_c_read_operations' 2024-09-11T05:08:45.751 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-11T05:08:45.752 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.752 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.752 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.752 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.752 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_c_read_operations 2024-09-11T05:08:45.752 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-11T05:08:45.754 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.757 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_c_read_operations 2024-09-11T05:08:45.757 INFO:tasks.workunit.client.0.smithi190.stdout:test api_c_read_operations on pid 28519 2024-09-11T05:08:45.758 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28519 2024-09-11T05:08:45.758 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-11T05:08:45.758 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_c_read_operations on pid 28519' 2024-09-11T05:08:45.758 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28519 2024-09-11T05:08:45.758 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.758 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.759 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-11T05:08:45.759 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.759 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.759 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.759 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.759 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-11T05:08:45.760 INFO:tasks.workunit.client.0.smithi190.stdout:src has 1..20) ==== 3534+0+0 (secure 0 0 0) 0x7f6d600ebce0 con 0x555a8cba9540 2024-09-11T05:08:45.760 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d57fff640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_map(20..20 src has 1..20) 2024-09-11T05:08:45.760 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d57fff640 3 client.4409.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T05:08:45.760 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d57fff640 3 client.4409.objecter handle_osd_map decoding full epoch 20 2024-09-11T05:08:45.760 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d57fff640 20 client.4409.objecter dump_active .. 0 homeless 2024-09-11T05:08:45.760 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 10 librados: wait_for_osdmap done waiting 2024-09-11T05:08:45.760 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d600ab070 con 0x555a8cba9540 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d6e0ef640 1 --2- 172.21.15.190:0/2305799564 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d5003b0f0 0x7f6d5003d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.650+0000 7f6d6e0ef640 10 monclient: get_auth_request con 0x7f6d5003b0f0 auth_method 0 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6d600bb410 con 0x555a8cba9540 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d57fff640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d6d0ed640 10 client.4409.objecter _get_latest_version latest 20, have it 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d70de6880 10 client.4409.objecter create_pool name=ceph_test_rados_api_asio 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d70de6880 10 client.4409.objecter pool_op_submit 1 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d70de6880 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x555a8caa5880 con 0x555a8cba9540 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d6e0ef640 1 --2- 172.21.15.190:0/2305799564 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d5003b0f0 0x7f6d5003d5b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x555a8cae6200 tx=0x7f6d58005a10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.654+0000 7f6d57fff640 10 client.4409.objecter ms_handle_connect 0x7f6d5003b0f0 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== mon.2 v2:172.21.15.138:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v21) ==== 43+0+0 (secure 0 0 0) 0x7f6d60017f80 con 0x555a8cba9540 2024-09-11T05:08:45.761 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 pool_op_reply(tid 1 (0) Success v21) 2024-09-11T05:08:45.762 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 10 client.4409.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v21) 2024-09-11T05:08:45.762 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 10 client.4409.objecter have request 1 at 0x555a8cb106d0 Op: create 2024-09-11T05:08:45.762 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 20 client.4409.objecter waiting for client to reach epoch 21 before calling back 2024-09-11T05:08:45.762 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-11T05:08:45.763 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-11T05:08:45.765 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-11T05:08:45.777 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s list_parallel 2024-09-11T05:08:45.777 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' list_parallel' 2024-09-11T05:08:45.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:08:45.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 10 monclient: _renew_subs 2024-09-11T05:08:45.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:08:45.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=21}) -- 0x7f6d50040510 con 0x555a8cba9540 2024-09-11T05:08:45.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 10 client.4409.objecter done 2024-09-11T05:08:45.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== mon.2 v2:172.21.15.138:3301/0 8 ==== osd_map(21..21 src has 1..21) ==== 3603+0+0 (secure 0 0 0) 0x7f6d60003c10 con 0x555a8cba9540 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_map(21..21 src has 1..21) 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 3 client.4409.objecter handle_osd_map got epochs [21,21] > 20 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 3 client.4409.objecter handle_osd_map decoding incremental epoch 21 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d57fff640 20 client.4409.objecter dump_active .. 0 homeless 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 10 librados: set snap read head -> 1 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 10 librados: writefull oid=exist nspace= 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cba38f0 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 1 --2- 172.21.15.190:0/2305799564 >> v2:172.21.15.138:6808/4002974512 conn(0x555a8cb0b980 0x555a8cc55800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:45.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 2 osd.1 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 2 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 15 client.4409.objecter _send_op 2 to 6.1 on osd.1 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:2 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x555a8cc55d40 con 0x555a8cb0b980 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.758+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.762+0000 7f6d6f0f1640 1 --2- 172.21.15.190:0/2305799564 >> v2:172.21.15.138:6808/4002974512 conn(0x555a8cb0b980 0x555a8cc55800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.762+0000 7f6d6f0f1640 10 monclient: get_auth_request con 0x555a8cb0b980 auth_method 0 2024-09-11T05:08:45.785 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.790 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo list_parallel 2024-09-11T05:08:45.792 INFO:tasks.workunit.client.0.smithi190.stdout: test list_parallel on pid 28561 2024-09-11T05:08:45.792 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=list_parallel 2024-09-11T05:08:45.792 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28561 2024-09-11T05:08:45.793 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test list_parallel on pid 28561' 2024-09-11T05:08:45.793 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28561 2024-09-11T05:08:45.793 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.793 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.793 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s open_pools_parallel 2024-09-11T05:08:45.793 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' open_pools_parallel' 2024-09-11T05:08:45.793 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.795 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo open_pools_parallel 2024-09-11T05:08:45.796 INFO:tasks.workunit.client.0.smithi190.stdout:test open_pools_parallel on pid 28577 2024-09-11T05:08:45.797 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=open_pools_parallel 2024-09-11T05:08:45.797 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28577 2024-09-11T05:08:45.797 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test open_pools_parallel on pid 28577' 2024-09-11T05:08:45.797 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28577 2024-09-11T05:08:45.797 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T05:08:45.797 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.797 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s delete_pools_parallel 2024-09-11T05:08:45.797 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' delete_pools_parallel' 2024-09-11T05:08:45.798 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-11T05:08:45.798 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo delete_pools_parallel 2024-09-11T05:08:45.799 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.799 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.799 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.799 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.799 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.799 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-11T05:08:45.800 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-11T05:08:45.802 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-11T05:08:45.804 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-11T05:08:45.806 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.806 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.806 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.806 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stdout:test delete_pools_parallel on pid 28608 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=delete_pools_parallel 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28608 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test delete_pools_parallel on pid 28608' 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28608 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.807 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.808 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ list_parallel: /' 2024-09-11T05:08:45.811 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s cls 2024-09-11T05:08:45.811 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' cls' 2024-09-11T05:08:45.812 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.817 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-11T05:08:45.818 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo cls 2024-09-11T05:08:45.819 INFO:tasks.workunit.client.0.smithi190.stdout:test cls on pid 28632 2024-09-11T05:08:45.820 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=cls 2024-09-11T05:08:45.820 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28632 2024-09-11T05:08:45.820 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test cls on pid 28632' 2024-09-11T05:08:45.820 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28632 2024-09-11T05:08:45.820 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.820 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.820 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s cmd 2024-09-11T05:08:45.820 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' cmd' 2024-09-11T05:08:45.821 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.821 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.822 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.822 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.822 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-11T05:08:45.824 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.824 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.824 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.824 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.829 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.831 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ cls: /' 2024-09-11T05:08:45.831 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_cls 2024-09-11T05:08:45.832 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo cmd 2024-09-11T05:08:45.832 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-11T05:08:45.834 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_cls.log 2024-09-11T05:08:45.836 INFO:tasks.workunit.client.0.smithi190.stdout:test cmd on pid 28658 2024-09-11T05:08:45.836 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=cmd 2024-09-11T05:08:45.836 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28658 2024-09-11T05:08:45.836 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test cmd on pid 28658' 2024-09-11T05:08:45.837 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28658 2024-09-11T05:08:45.837 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.837 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.837 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-11T05:08:45.837 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-11T05:08:45.844 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s handler_error 2024-09-11T05:08:45.845 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' handler_error' 2024-09-11T05:08:45.845 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-11T05:08:45.846 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.847 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.847 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.847 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.854 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_cmd 2024-09-11T05:08:45.854 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-11T05:08:45.861 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo handler_error 2024-09-11T05:08:45.864 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.866 INFO:tasks.workunit.client.0.smithi190.stdout:test handler_error on pid 28705 2024-09-11T05:08:45.867 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=handler_error 2024-09-11T05:08:45.867 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28705 2024-09-11T05:08:45.867 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test handler_error on pid 28705' 2024-09-11T05:08:45.867 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28705 2024-09-11T05:08:45.867 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.867 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.877 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s io 2024-09-11T05:08:45.878 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' io' 2024-09-11T05:08:45.878 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-11T05:08:45.879 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ cmd: /' 2024-09-11T05:08:45.879 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.879 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.879 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.879 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.879 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo io 2024-09-11T05:08:45.880 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.880 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-11T05:08:45.885 INFO:tasks.workunit.client.0.smithi190.stdout:test io on pid 28772 2024-09-11T05:08:45.885 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=io 2024-09-11T05:08:45.885 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28772 2024-09-11T05:08:45.885 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test io on pid 28772' 2024-09-11T05:08:45.885 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28772 2024-09-11T05:08:45.885 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.885 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.886 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_handler_error 2024-09-11T05:08:45.888 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s ec_io 2024-09-11T05:08:45.888 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' ec_io' 2024-09-11T05:08:45.889 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-11T05:08:45.890 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.891 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.892 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.892 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.892 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.892 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ handler_error: /' 2024-09-11T05:08:45.893 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo ec_io 2024-09-11T05:08:45.893 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_io 2024-09-11T05:08:45.895 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_io.log 2024-09-11T05:08:45.895 INFO:tasks.workunit.client.0.smithi190.stdout:test ec_io on pid 28803 2024-09-11T05:08:45.896 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=ec_io 2024-09-11T05:08:45.896 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28803 2024-09-11T05:08:45.896 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test ec_io on pid 28803' 2024-09-11T05:08:45.896 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28803 2024-09-11T05:08:45.896 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.896 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.896 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-11T05:08:45.896 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ io: /' 2024-09-11T05:08:45.897 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.897 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.897 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s list 2024-09-11T05:08:45.898 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.898 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.898 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' list' 2024-09-11T05:08:45.898 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-11T05:08:45.898 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ ec_io: /' 2024-09-11T05:08:45.901 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_ec_io 2024-09-11T05:08:45.903 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo list 2024-09-11T05:08:45.903 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.906 INFO:tasks.workunit.client.0.smithi190.stdout:test list on pid 28830 2024-09-11T05:08:45.906 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=list 2024-09-11T05:08:45.907 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28830 2024-09-11T05:08:45.907 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test list on pid 28830' 2024-09-11T05:08:45.907 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28830 2024-09-11T05:08:45.907 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.907 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.907 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-11T05:08:45.907 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s ec_list 2024-09-11T05:08:45.907 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' ec_list' 2024-09-11T05:08:45.912 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.912 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.912 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.912 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.912 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_list 2024-09-11T05:08:45.913 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_list.log 2024-09-11T05:08:45.913 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ list: /' 2024-09-11T05:08:45.920 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.928 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo ec_list 2024-09-11T05:08:45.929 INFO:tasks.workunit.client.0.smithi190.stdout:test ec_list on pid 28910 2024-09-11T05:08:45.929 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=ec_list 2024-09-11T05:08:45.929 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28910 2024-09-11T05:08:45.930 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test ec_list on pid 28910' 2024-09-11T05:08:45.930 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28910 2024-09-11T05:08:45.930 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.930 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-11T05:08:45.930 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.930 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s misc 2024-09-11T05:08:45.931 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' misc' 2024-09-11T05:08:45.931 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.931 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.931 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.931 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.931 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_ec_list 2024-09-11T05:08:45.931 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-11T05:08:45.931 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.932 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo misc 2024-09-11T05:08:45.932 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ ec_list: /' 2024-09-11T05:08:45.933 INFO:tasks.workunit.client.0.smithi190.stdout:test misc on pid 28927 2024-09-11T05:08:45.933 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=misc 2024-09-11T05:08:45.934 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28927 2024-09-11T05:08:45.934 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test misc on pid 28927' 2024-09-11T05:08:45.934 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28927 2024-09-11T05:08:45.934 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.934 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.934 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s pool 2024-09-11T05:08:45.935 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' pool' 2024-09-11T05:08:45.935 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo pool 2024-09-11T05:08:45.935 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-11T05:08:45.935 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.936 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.936 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.937 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.937 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.937 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_misc 2024-09-11T05:08:45.937 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_misc.log 2024-09-11T05:08:45.937 INFO:tasks.workunit.client.0.smithi190.stdout:test pool on pid 28937 2024-09-11T05:08:45.937 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=pool 2024-09-11T05:08:45.938 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ misc: /' 2024-09-11T05:08:45.938 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28937 2024-09-11T05:08:45.938 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test pool on pid 28937' 2024-09-11T05:08:45.938 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28937 2024-09-11T05:08:45.938 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.938 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.938 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-11T05:08:45.939 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.939 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.939 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.939 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.939 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_pool 2024-09-11T05:08:45.939 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ pool: /' 2024-09-11T05:08:45.941 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s read_operations 2024-09-11T05:08:45.941 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' read_operations' 2024-09-11T05:08:45.942 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo read_operations 2024-09-11T05:08:45.942 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.946 INFO:tasks.workunit.client.0.smithi190.stdout:test read_operations on pid 28961 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=read_operations 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28961 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test read_operations on pid 28961' 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28961 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s snapshots 2024-09-11T05:08:45.947 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' snapshots' 2024-09-11T05:08:45.948 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo snapshots 2024-09-11T05:08:45.948 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.948 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.948 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.949 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.949 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.949 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_read_operations 2024-09-11T05:08:45.949 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ read_operations: /' 2024-09-11T05:08:45.952 INFO:tasks.workunit.client.0.smithi190.stdout:test snapshots on pid 28990 2024-09-11T05:08:45.953 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=snapshots 2024-09-11T05:08:45.953 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28990 2024-09-11T05:08:45.953 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test snapshots on pid 28990' 2024-09-11T05:08:45.953 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28990 2024-09-11T05:08:45.953 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.953 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.953 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s watch_notify 2024-09-11T05:08:45.953 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' watch_notify' 2024-09-11T05:08:45.954 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_pool.log 2024-09-11T05:08:45.955 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-11T05:08:45.955 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-11T05:08:45.957 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.957 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo watch_notify 2024-09-11T05:08:45.958 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.958 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.959 INFO:tasks.workunit.client.0.smithi190.stdout:test watch_notify on pid 29014 2024-09-11T05:08:45.960 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=watch_notify 2024-09-11T05:08:45.960 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29014 2024-09-11T05:08:45.960 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test watch_notify on pid 29014' 2024-09-11T05:08:45.960 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=29014 2024-09-11T05:08:45.960 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T05:08:45.960 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.960 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.960 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.961 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-11T05:08:45.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_snapshots 2024-09-11T05:08:45.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-11T05:08:45.963 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ snapshots: /' 2024-09-11T05:08:45.963 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.963 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.963 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.963 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-11T05:08:45.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ watch_notify: /' 2024-09-11T05:08:45.966 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s write_operations 2024-09-11T05:08:45.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' write_operations' 2024-09-11T05:08:45.966 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo write_operations 2024-09-11T05:08:45.968 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_watch_notify 2024-09-11T05:08:45.968 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T05:08:45.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=write_operations 2024-09-11T05:08:45.971 INFO:tasks.workunit.client.0.smithi190.stdout:test write_operations on pid 29044 2024-09-11T05:08:45.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29044 2024-09-11T05:08:45.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test write_operations on pid 29044' 2024-09-11T05:08:45.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=29044 2024-09-11T05:08:45.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-11T05:08:45.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ ret=0 2024-09-11T05:08:45.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T05:08:45.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:08:45.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28180 2024-09-11T05:08:45.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28180 2024-09-11T05:08:45.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T05:08:45.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T05:08:45.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T05:08:45.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_write_operations 2024-09-11T05:08:45.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-11T05:08:45.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ write_operations: /' 2024-09-11T05:08:47.004 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.782+0000 7f6d57fff640 10 client.4409.objecter ms_handle_connect 0x555a8cb0b980 2024-09-11T05:08:47.004 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:45.782+0000 7f6d6f0f1640 1 --2- 172.21.15.190:0/2305799564 >> v2:172.21.15.138:6808/4002974512 conn(0x555a8cb0b980 0x555a8cc55800 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:08:47.004 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 1 ==== osd_map(22..22 src has 1..22) ==== 12910+0+0 (crc 0 0 0) 0x7f6d64005b10 con 0x555a8cb0b980 2024-09-11T05:08:47.004 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_map(22..22 src has 1..22) 2024-09-11T05:08:47.004 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 3 client.4409.objecter handle_osd_map got epochs [22,22] > 21 2024-09-11T05:08:47.004 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 3 client.4409.objecter handle_osd_map decoding incremental epoch 22 2024-09-11T05:08:47.004 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 10 client.4409.objecter checking op 2 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 20 client.4409.objecter dump_active .. 0 homeless 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:46.994+0000 7f6d57fff640 20 client.4409.objecter 2 6.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 2 ==== osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 2 ondisk uv 1 in 6.1 attempt 0 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 2 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 2 2024-09-11T05:08:47.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 2 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 10 librados: Objecter returned from writefull r=0 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cba38f0 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 3 osd.1 2024-09-11T05:08:47.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 3 2024-09-11T05:08:47.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 15 client.4409.objecter _send_op 3 to 6.1 on osd.1 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:3 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cb0d500 con 0x555a8cb0b980 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cb0d8d0 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 4 osd.1 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 4 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 15 client.4409.objecter _send_op 4 to 6.1 on osd.1 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:4 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cb0dd80 con 0x555a8cb0b980 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.002+0000 7f6d70de6880 5 client.4409.objecter 2 in flight 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 3 ondisk uv 1 in 6.1 attempt 0 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 5 2024-09-11T05:08:47.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 3 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 3 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 3 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 4 ondisk uv 0 in 6.1 attempt 0 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 4 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 4 2024-09-11T05:08:47.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 4 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (5 ms) 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cb0d8d0 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 5 osd.1 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 5 2024-09-11T05:08:47.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _send_op 5 to 6.1 on osd.1 2024-09-11T05:08:47.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:5 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cb0e160 con 0x555a8cb0b980 2024-09-11T05:08:47.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc560e0 2024-09-11T05:08:47.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 6 osd.1 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 6 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _send_op 6 to 6.1 on osd.1 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:6 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cba3aa0 con 0x555a8cb0b980 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 5 client.4409.objecter 2 in flight 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-11T05:08:47.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 5 ondisk uv 1 in 6.1 attempt 0 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 5 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 5 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 5 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 5 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 6 ondisk uv 0 in 6.1 attempt 0 2024-09-11T05:08:47.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T05:08:47.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 6 2024-09-11T05:08:47.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 6 2024-09-11T05:08:47.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 6 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cb0d8d0 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.018 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 7 osd.1 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 7 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _send_op 7 to 6.1 on osd.1 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:7 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc561f0 con 0x555a8cb0b980 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc96710 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 8 osd.1 2024-09-11T05:08:47.019 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 8 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _send_op 8 to 6.1 on osd.1 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:8 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc96bc0 con 0x555a8cb0b980 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 5 client.4409.objecter 2 in flight 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 7 ondisk uv 1 in 6.1 attempt 0 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 5 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 7 2024-09-11T05:08:47.020 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 7 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 7 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 8 ondisk uv 0 in 6.1 attempt 0 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 8 2024-09-11T05:08:47.021 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 8 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 8 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 librados: queue_aio_write 0x555a8ca6a530 completion 0x555a8cc96710 write_seq 1 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cb0d8d0 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.022 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 9 osd.1 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 9 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 15 client.4409.objecter _send_op 9 to 6.1 on osd.1 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:9 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc96820 con 0x555a8cb0b980 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.006+0000 7f6d70de6880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 9 ==== osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) 2024-09-11T05:08:47.023 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 9 ondisk uv 2 in 6.1 attempt 0 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 9 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 9 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 9 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d6f0f1640 20 librados: complete_aio_write 0x555a8cc96710 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (4 ms) 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-11T05:08:47.024 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 20 librados: queue_aio_write 0x555a8ca6a530 completion 0x555a8cc575f0 write_seq 2 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cb0d8d0 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 10 osd.1 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 10 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 15 client.4409.objecter _send_op 10 to 6.1 on osd.1 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:10 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc57760 con 0x555a8cb0b980 2024-09-11T05:08:47.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.010+0000 7f6d70de6880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 10 ==== osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 10 ondisk uv 3 in 6.1 attempt 0 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 10 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 10 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 10 2024-09-11T05:08:47.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d6f0f1640 20 librados: complete_aio_write 0x555a8cc575f0 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (2 ms) 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 20 librados: queue_aio_write 0x555a8ca6a530 completion 0x555a8cb0d8d0 write_seq 3 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc79a30 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 11 osd.1 2024-09-11T05:08:47.027 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.028 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 11 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 15 client.4409.objecter _send_op 11 to 6.1 on osd.1 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:11 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x555a8cb0d9e0 con 0x555a8cb0b980 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.014+0000 7f6d70de6880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 11 ==== osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 11 ondisk uv 4 in 6.1 attempt 0 2024-09-11T05:08:47.029 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.030 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 11 2024-09-11T05:08:47.030 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 11 2024-09-11T05:08:47.030 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.030 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 11 2024-09-11T05:08:47.030 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.030 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 20 librados: complete_aio_write 0x555a8cb0d8d0 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (4 ms) 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc565e0 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 12 osd.1 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 12 2024-09-11T05:08:47.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _send_op 12 to 6.1 on osd.1 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:12 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cb0d9e0 con 0x555a8cb0b980 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc56ba0 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 13 osd.1 2024-09-11T05:08:47.032 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.033 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 13 2024-09-11T05:08:47.033 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _send_op 13 to 6.1 on osd.1 2024-09-11T05:08:47.033 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:13 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc57050 con 0x555a8cb0b980 2024-09-11T05:08:47.033 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.033 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 5 client.4409.objecter 2 in flight 2024-09-11T05:08:47.033 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 12 ondisk uv 4 in 6.1 attempt 0 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 5 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 12 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 12 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 12 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.034 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 13 ondisk uv 0 in 6.1 attempt 0 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 13 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 13 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 13 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-11T05:08:47.035 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc565e0 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 14 osd.1 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 14 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _send_op 14 to 6.1 on osd.1 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:14 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cb0d9e0 con 0x555a8cb0b980 2024-09-11T05:08:47.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc56d30 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 15 osd.1 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 15 2024-09-11T05:08:47.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _send_op 15 to 6.1 on osd.1 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:15 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc7a1b0 con 0x555a8cb0b980 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 5 client.4409.objecter 2 in flight 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 14 ondisk uv 4 in 6.1 attempt 0 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 5 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 14 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 14 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 14 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 15 ondisk uv 0 in 6.1 attempt 0 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 15 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 15 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 15 2024-09-11T05:08:47.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc79a30 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 16 osd.1 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 16 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _send_op 16 to 6.1 on osd.1 2024-09-11T05:08:47.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:16 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc7a1b0 con 0x555a8cb0b980 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc565e0 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 17 osd.1 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 17 2024-09-11T05:08:47.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _send_op 17 to 6.1 on osd.1 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:17 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc56d30 con 0x555a8cb0b980 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 5 client.4409.objecter 2 in flight 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 16 ondisk uv 4 in 6.1 attempt 0 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 5 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 16 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 16 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 5 2024-09-11T05:08:47.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 16 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 17 ondisk uv 0 in 6.1 attempt 0 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 17 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 17 2024-09-11T05:08:47.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 17 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 librados: queue_aio_write 0x555a8ca6a530 completion 0x555a8cb0daf0 write_seq 4 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc565e0 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 18 osd.1 2024-09-11T05:08:47.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 18 2024-09-11T05:08:47.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 15 client.4409.objecter _send_op 18 to 6.1 on osd.1 2024-09-11T05:08:47.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:18 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc7a1b0 con 0x555a8cb0b980 2024-09-11T05:08:47.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.018+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 18 ==== osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) 2024-09-11T05:08:47.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 18 ondisk uv 5 in 6.1 attempt 0 2024-09-11T05:08:47.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (5 ms) 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 18 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 18 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 18 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 20 librados: complete_aio_write 0x555a8cb0daf0 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 librados: queue_aio_write 0x555a8ca6a530 completion 0x555a8cb0dc50 write_seq 5 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc565e0 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 19 osd.1 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 19 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 15 client.4409.objecter _send_op 19 to 6.1 on osd.1 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:19 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc56d30 con 0x555a8cb0b980 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 19 ==== osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 19 ondisk uv 6 in 6.1 attempt 0 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 19 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 19 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 19 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d6f0f1640 20 librados: complete_aio_write 0x555a8cb0dc50 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 librados: queue_aio_write 0x555a8ca6a530 completion 0x555a8cc57120 write_seq 6 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 10 client.4409.objecter _op_submit op 0x555a8cc565e0 2024-09-11T05:08:47.049 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 10 client.4409.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter _get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 10 client.4409.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 20 osd.1 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter get_session s=0x555a8cb0c8f0 osd=1 3 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 15 client.4409.objecter _session_op_assign 1 20 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 15 client.4409.objecter _send_op 20 to 6.1 on osd.1 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 --> v2:172.21.15.138:6808/4002974512 -- osd_op(unknown.0.0:20 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x555a8cc7a1b0 con 0x555a8cb0b980 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.026+0000 7f6d70de6880 5 client.4409.objecter 1 in flight 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 <== osd.1 v2:172.21.15.138:6808/4002974512 20 ==== osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f6d640063b0 con 0x555a8cb0b980 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 10 client.4409.objecter ms_dispatch 0x555a8ca74f00 osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) 2024-09-11T05:08:47.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 10 client.4409.objecter in handle_osd_op_reply 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 7 client.4409.objecter handle_osd_op_reply 20 ondisk uv 7 in 6.1 attempt 0 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 10 client.4409.objecter op 0 rval 0 len 0 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 15 client.4409.objecter handle_osd_op_reply completed tid 20 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 15 client.4409.objecter _finish_op 20 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 4 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 15 client.4409.objecter _session_op_remove 1 20 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 5 client.4409.objecter 0 in flight 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3 ms) 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 20 librados: complete_aio_write 0x555a8cc57120 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 10 librados: watch_flush enter 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 10 librados: watch_flush exit 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 10 client.4409.objecter close_session for osd.1 2024-09-11T05:08:47.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 >> v2:172.21.15.138:6808/4002974512 conn(0x555a8cb0b980 msgr2=0x555a8cc55800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 --2- 172.21.15.190:0/2305799564 >> v2:172.21.15.138:6808/4002974512 conn(0x555a8cb0b980 0x555a8cc55800 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 20 client.4409.objecter put_session s=0x555a8cb0c8f0 osd=1 1 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 20 client.4409.objecter shutdown clearing up homeless session... 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 10 client.4409.objecter successfully canceled tick 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d5003b0f0 msgr2=0x7f6d5003d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 --2- 172.21.15.190:0/2305799564 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d5003b0f0 0x7f6d5003d5b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x555a8cae6200 tx=0x7f6d58005a10 comp rx=0 tx=0).stop 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 10 monclient: shutdown 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cba9540 msgr2=0x555a8cc3f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 --2- 172.21.15.190:0/2305799564 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x555a8cba9540 0x555a8cc3f3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d60001470 tx=0x7f6d5c004130 comp rx=0 tx=0).stop 2024-09-11T05:08:47.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d6f0f1640 1 -- 172.21.15.190:0/2305799564 reap_dead start 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 shutdown_connections 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 >> 172.21.15.190:0/2305799564 conn(0x555a8caf8480 msgr2=0x555a8cb163a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 shutdown_connections 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 -- 172.21.15.190:0/2305799564 wait complete. 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.030+0000 7f6d70de6880 1 librados: shutdown 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [----------] 12 tests from AsioRados (29 ms total) 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [----------] Global test environment tear-down 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1398 ms total) 2024-09-11T05:08:47.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] Global test environment set-up. 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjects (245 ms) 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (35 ms) 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo1,foo2,foo3 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo1 2024-09-11T05:08:47.428 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo2 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo3 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo1,foo4,foo5 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo4 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo5 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo1 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo6,foo7 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo7 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo6 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo4 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo5 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns2:foo7 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns2:foo6 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo1 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo1 2024-09-11T05:08:47.429 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo2 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo3 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (89 ms) 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 1 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 10 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 13 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 7 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 14 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 0 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 15 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 11 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 5 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 8 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 6 0 2024-09-11T05:08:47.430 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 3 0 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 4 0 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 12 0 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 9 0 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2 0 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (107 ms) 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: x cursor=MIN 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-09-11T05:08:47.431 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-09-11T05:08:47.432 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:566253c9:::13:head 2024-09-11T05:08:47.433 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-09-11T05:08:47.434 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-09-11T05:08:47.434 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-09-11T05:08:47.434 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-09-11T05:08:47.434 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2024-09-11T05:08:47.577 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:863748b0:::15:head 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-09-11T05:08:47.578 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-09-11T05:08:47.579 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:e960b815:::9:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-09-11T05:08:47.580 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=1 expected=1 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=11 expected=11 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:863748b0:::15:head 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=15 expected=15 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-09-11T05:08:47.581 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=14 expected=14 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=7 expected=7 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=10 expected=10 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=0 expected=0 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-09-11T05:08:47.582 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-09-11T05:08:47.583 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-09-11T05:08:47.583 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=2 expected=2 2024-09-11T05:08:47.583 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:e960b815:::9:head 2024-09-11T05:08:48.820 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: Running main() from gmock_main.cc 2024-09-11T05:08:48.820 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [----------] Global test environment set-up. 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2913 ms) 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [----------] 1 test from neocls_handler_error (2913 ms total) 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [----------] Global test environment tear-down 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2913 ms total) 2024-09-11T05:08:48.821 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [ PASSED ] 1 test. 2024-09-11T05:08:48.831 INFO:tasks.workunit.client.0.smithi190.stdout: cls: Running main() from gmock_main.cc 2024-09-11T05:08:48.831 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [----------] Global test environment set-up. 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [----------] 1 test from NeoRadosCls 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [ OK ] NeoRadosCls.DNE (2968 ms) 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [----------] 1 test from NeoRadosCls (2968 ms total) 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [----------] Global test environment tear-down 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [==========] 1 test from 1 test suite ran. (2968 ms total) 2024-09-11T05:08:48.832 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [ PASSED ] 1 test. 2024-09-11T05:08:48.841 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-11T05:08:48.841 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-11T05:08:48.841 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-11T05:08:48.841 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-11T05:08:48.841 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-11T05:08:48.841 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (97 ms) 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (59 ms) 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2940 ms) 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3096 ms total) 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3096 ms total) 2024-09-11T05:08:48.842 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-11T05:08:48.846 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (277 ms) 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (38 ms) 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (14 ms) 2024-09-11T05:08:48.847 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (28 ms) 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (12 ms) 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (4 ms) 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (10 ms) 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (3 ms) 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (3 ms) 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (4 ms) 2024-09-11T05:08:48.848 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (14 ms) 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (18 ms) 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (15 ms) 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (32 ms) 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (7 ms) 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (484 ms total) 2024-09-11T05:08:48.849 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: 2024-09-11T05:08:48.850 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-11T05:08:48.850 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (3030 ms total) 2024-09-11T05:08:48.850 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-11T05:08:48.980 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-11T05:08:48.980 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-11T05:08:48.980 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (107 ms) 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (74 ms) 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2901 ms) 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.794094+0000 mon.b [INF] from='client.? 172.21.15.190:0/732205744' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi190-28311-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.794211+0000 mon.b [INF] from='client.? 172.21.15.190:0/10400619' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28439-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.794330+0000 mon.b [INF] from='client.? 172.21.15.190:0/4165574098' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi190-28049-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.794409+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi190-28089-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.981 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.794525+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi190-28280-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.982 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.794645+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi190-28332-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.982 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.794792+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28444-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.982 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.794938+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28516-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.982 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.795033+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi190-28245-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.982 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.795110+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi190-28303-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.982 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.795168+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi190-28522-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:48.982 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.849933+0000 mon.b [INF] from='client.? 172.21.15.190:0/1811424013' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28187-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T05:08:49.553 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.930884+0000 mon.c [INF] from='client.? 172.21.15.190:0/869558095' entity='clie list_parallel: process_1_[28636]: starting. 2024-09-11T05:08:49.553 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28636]: creating pool ceph_test_rados_list_parallel.smithi190-28606 2024-09-11T05:08:49.553 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28636]: created object 0... 2024-09-11T05:08:49.554 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28636]: created object 25... 2024-09-11T05:08:49.554 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28636]: created object 49... 2024-09-11T05:08:49.554 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28636]: finishing. 2024-09-11T05:08:49.554 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28636]: shutting down. 2024-09-11T05:08:49.554 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28638]: starting. 2024-09-11T05:08:49.554 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28638]: listing objects. 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28638]: listed object 0... 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28638]: listed object 25... 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28638]: saw 50 objects 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28638]: shutting down. 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29327]: starting. 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29327]: creating pool ceph_test_rados_list_parallel.smithi190-28606 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29327]: created object 0... 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29327]: created object 25... 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29327]: created object 49... 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29327]: finishing. 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29327]: shutting down. 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29328]: starting. 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29328]: listing objects. 2024-09-11T05:08:49.555 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29328]: listed object 0... 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29328]: listed object 25... 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29328]: saw 41 objects 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29328]: shutting down. 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29329]: starting. 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29329]: removed 25 objects... 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29329]: removed half of the objects 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29329]: removed 50 objects... 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29329]: removed 50 objects 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29329]: shutting down. 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29394]: starting. 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29394]: creating pool ceph_test_rados_list_parallel.smithi190-28606 2024-09-11T05:08:49.556 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29394]: created object 0... 2024-09-11T05:08:49.557 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29394]: created object 25... 2024-09-11T05:08:49.557 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29394]: created object 49... 2024-09-11T05:08:49.557 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29394]: finishing. 2024-09-11T05:08:49.557 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29394]: shutting down. 2024-09-11T05:08:49.557 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.557 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.557 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29395]: starting. 2024-09-11T05:08:49.557 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29395]: listing objects. 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29395]: listed object 0... 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29395]: listed object 25... 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29395]: listed object 50... 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29395]: saw 55 objects 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29395]: shutting down. 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29396]: starting. 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29396]: added 25 objects... 2024-09-11T05:08:49.558 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29396]: added half of the objects 2024-09-11T05:08:49.559 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29396]: added 50 objects... 2024-09-11T05:08:49.559 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29396]: added 50 objects 2024-09-11T05:08:49.559 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29396]: shutting down. 2024-09-11T05:08:49.559 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.559 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:49.859 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: **********************nt.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi190-28065-24"} : dispatch 2024-09-11T05:08:49.859 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:47.970625+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi190-28065-24"} : dispatch 2024-09-11T05:08:49.859 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.006514+0000 mon.c [INF] from='client.? 172.21.15.190:0/869558095' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi190-28065-24"} : dispatch 2024-09-11T05:08:49.859 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.026518+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi190-28065-24"} : dispatch 2024-09-11T05:08:49.859 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.036926+0000 mon.c [INF] from='client.? 172.21.15.190:0/869558095' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi190-28065-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T05:08:49.859 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.044623+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi190-28065-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.755759+0000 mon.b [INF] from='client.? 172.21.15.190:0/869891065' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi190-28057-16"} : dispatch 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.756657+0000 mon.b [INF] from='client.? 172.21.15.190:0/869891065' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi190-28057-16"} : dispatch 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.757191+0000 mon.b [INF] from='client.? 172.21.15.190:0/869891065' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi190-28057-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.804645+0000 mon.b [INF] from='client.? 172.21.15.190:0/2813022488' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi190-28356-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi190-28356-1"}]': finished 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.804726+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi190-28810-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi190-28810-1"}]': finished 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.804806+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi190-28915-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi190-28915-1"}]': finished 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.804963+0000 mon.b [INF] from='client.? 172.21.15.190:0/1811424013' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28187-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.805143+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi190-28065-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.805229+0000 mon.b [INF] from='client.? 172.21.15.190:0/869891065' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi190-28057-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.815376+0000 mon.c [INF] from='client.? 172.21.15.190:0/869558095' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi190-28065-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi190-28065-24"} : dispatch 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.828694+0000 mon.b [INF] from='client.? 172.21.15.190:0/869891065' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi190-28057-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi190-28057-16"} : dispatch 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.829220+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi190-28065-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi190-28065-24"} : dispatch 2024-09-11T05:08:49.860 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.842191+0000 mon.b [INF] from='client.? 172.21.15.190:0/3976286268' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28235-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.842387+0000 mon.b [INF] from='client.? 172.21.15.190:0/4291422535' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi190-28041-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.929019+0000 mon.c [INF] from='client.? 172.21.15.190:0/3472994114' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi190-28311-7"} : dispatch 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.980159+0000 mon.c [INF] from='client.? 172.21.15.190:0/17437294' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.983221+0000 client.admin [INF] onexx 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.995545+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi190-28311-7"} : dispatch 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.995872+0000 mon.b [INF] from='client.? 172.21.15.190:0/3824597767' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi190-28303-7"} : dispatch 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:48.997317+0000 mon.c [INF] from='client.? 172.21.15.190:0/3472994114' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi190-28311-7"} : dispatch 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:49.015150+0000 mon.b [INF] from='client.? 172.21.15.190:0/3824597767' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi190-28303-7"} : dispatch 2024-09-11T05:08:49.861 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:49.016078+0000 mon.b [INF] from='client.? 172.21.15.190:0/3824597767' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi190-28303-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T05:08:49.862 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:49.017225+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-11T05:08:49.862 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:49.026590+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi190-28311-7"} : dispatch 2024-09-11T05:08:49.862 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:49.030248+0000 mon.c [INF] from='client.? 172.21.15.190:0/3472994114' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi190-28311-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T05:08:51.139 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08********* 2024-09-11T05:08:51.139 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29644]: starting. 2024-09-11T05:08:51.139 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29644]: creating pool ceph_test_rados_list_parallel.smithi190-28606 2024-09-11T05:08:51.139 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29644]: created object 0... 2024-09-11T05:08:51.139 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29644]: created object 25... 2024-09-11T05:08:51.139 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29644]: created object 49... 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29644]: finishing. 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29644]: shutting down. 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: starting. 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: listing objects. 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: listed object 0... 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: listed object 25... 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: listed object 50... 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: listed object 75... 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: listed object 100... 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: saw 106 objects 2024-09-11T05:08:51.140 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29645]: shutting down. 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29646]: starting. 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29646]: added 25 objects... 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29646]: added half of the objects 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29646]: added 50 objects... 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29646]: added 50 objects 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29646]: shutting down. 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.141 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29647]: starting. 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29647]: added 25 objects... 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29647]: added half of the objects 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29647]: added 50 objects... 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29647]: added 50 objects 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29647]: shutting down. 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29648]: starting. 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29648]: removed 25 objects... 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29648]: removed half of the objects 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29648]: removed 50 objects... 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29648]: removed 50 objects 2024-09-11T05:08:51.142 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29648]: shutting down. 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29862]: starting. 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29862]: creating pool ceph_test_rados_list_parallel.smithi190-28606 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29862]: created object 0... 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29862]: created object 25... 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29862]: created object 49... 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29862]: finishing. 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29862]: shutting down. 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.143 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.144 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.144 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: starting. 2024-09-11T05:08:51.144 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: listing objects. 2024-09-11T05:08:51.864 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: listed object 0... 2024-09-11T05:08:51.864 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: listed object 25... 2024-09-11T05:08:51.864 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: listed object 50... 2024-09-11T05:08:51.864 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: listed object 75... 2024-09-11T05:08:51.864 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: listed object 100... 2024-09-11T05:08:51.864 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: listed object 125... 2024-09-11T05:08:51.864 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: saw 150 objects 2024-09-11T05:08:51.864 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29863]: shutting down. 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29864]: starting. 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29864]: added 25 objects... 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29864]: added half of the objects 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29864]: added 50 objects... 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29864]: added 50 objects 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29864]: shutting down. 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.865 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.866 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.866 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T05:08:51.866 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******* SUCCESS ********** 2024-09-11T05:08:52.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-11T05:08:52.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: seed 28065 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (1 ms) 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (200 ms) 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (12 ms) 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (9 ms) 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-09-11T05:08:52.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (496 ms) 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (6 ms) 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (4 ms) 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (10 ms) 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (7 ms) 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-09-11T05:08:52.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (8 ms) 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (5 ms) 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (22 ms) 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (9 ms) 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (5 ms) 2024-09-11T05:08:52.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (7 ms) 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (834 ms total) 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1185 ms) 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (25 ms) 2024-09-11T05:08:52.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-11T05:08:52.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (13 ms) 2024-09-11T05:08:52.872 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: Running main() from gmock_main.cc 2024-09-11T05:08:52.872 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-11T05:08:52.872 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [----------] Global test environment set-up. 2024-09-11T05:08:52.872 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-11T05:08:52.872 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1966 ms) 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2007 ms) 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3020 ms) 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6993 ms total) 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [----------] Global test environment tear-down 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6993 ms total) 2024-09-11T05:08:52.873 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ PASSED ] 3 tests. 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28661]: starting. 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28661]: creating pool ceph_test_rados_open_pools_parallel.smithi190-28591 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28661]: created object 0... 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28661]: created object 25... 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28661]: created object 49... 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28661]: finishing. 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28661]: shutting down. 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_2_[28662]: starting. 2024-09-11T05:08:52.885 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_2_[28662]: rados_pool_create. 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_2_[28662]: rados_ioctx_create. 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_2_[28662]: shutting down. 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******************************* 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29422]: starting. 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29422]: creating pool ceph_test_rados_open_pools_parallel.smithi190-28591 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29422]: created object 0... 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29422]: created object 25... 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29422]: created object 49... 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29422]: finishing. 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29422]: shutting down. 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******************************* 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_4_[29423]: starting. 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_4_[29423]: rados_pool_create. 2024-09-11T05:08:52.886 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_4_[29423]: rados_ioctx_create. 2024-09-11T05:08:52.887 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_4_[29423]: shutting down. 2024-09-11T05:08:52.887 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******************************* 2024-09-11T05:08:52.887 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******************************* 2024-09-11T05:08:52.887 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-11T05:08:52.915 INFO:tasks.workunit.client.0.smithi190.stdout::49.038954+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi190-28311-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T05:08:52.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:51.894505+0000 mon.b [INF] from='client.? 172.21.15.190:0/686815323' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi190-28810-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T05:08:52.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.203618+0000 mon.c [INF] from='client.? 172.21.15.190:0/869558095' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi190-28065-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-11T05:08:52.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.205060+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi190-28065-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-11T05:08:52.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.277330+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.0"} : dispatch 2024-09-11T05:08:52.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.278060+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.1"} : dispatch 2024-09-11T05:08:52.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.278553+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.2"} : dispatch 2024-09-11T05:08:52.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.279094+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.3"} : dispatch 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.279604+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.4"} : dispatch 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.280212+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.5"} : dispatch 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.280723+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.6"} : dispatch 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.281256+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.7"} : dispatch 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.281761+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.8"} : dispatch 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.282233+0000 mon.b [INF] from='client.4439 172.21.15.190:0/1559768584' entity='client.admin' cmd={"prefix": "pg deep-scrub", "pgid": "9.9"} : dispatch 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.840291+0000 mon.b [INF] from='client.4739 172.21.15.190:0/1531307463' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi190-28094-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi190-28094-10"}]': finished 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.840460+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi190-28915-1"}]': finished 2024-09-11T05:08:52.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.840584+0000 mon.b [INF] from='client.? 172.21.15.190:0/2813022488' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi190-28356-1"}]': finished 2024-09-11T05:08:53.898 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.840682+0000 mon.b [INF] from='client.? 172.21.15.190:0/1811424013' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-sm delete_pools_parallel: process_1_[28683]: starting. 2024-09-11T05:08:53.898 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28683]: creating pool ceph_test_rados_delete_pools_parallel.smithi190-28637 2024-09-11T05:08:53.898 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28683]: created object 0... 2024-09-11T05:08:53.898 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28683]: created object 25... 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28683]: created object 49... 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28683]: finishing. 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28683]: shutting down. 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_2_[28685]: starting. 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_2_[28685]: deleting pool ceph_test_rados_delete_pools_parallel.smithi190-28637 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_2_[28685]: shutting down. 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29735]: starting. 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29735]: creating pool ceph_test_rados_delete_pools_parallel.smithi190-28637 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29735]: created object 0... 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29735]: created object 25... 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29735]: created object 49... 2024-09-11T05:08:53.899 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29735]: finishing. 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29735]: shutting down. 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29737]: starting. 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29737]: listing objects. 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29737]: listed object 0... 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29737]: listed object 25... 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29737]: saw 50 objects 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29737]: shutting down. 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_4_[29736]: starting. 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_4_[29736]: deleting pool ceph_test_rados_delete_pools_parallel.smithi190-28637 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_4_[29736]: shutting down. 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T05:08:53.900 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T05:08:53.901 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-11T05:08:54.025 INFO:tasks.workunit.client.0.smithi190.stdout:ithi190-28187-4", "overlaypool": "test-rados-api-smithi190-28187-4-cache"}]': finished 2024-09-11T05:08:54.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.840785+0000 mon.b [INF] from='client.? 172.21.15.190:0/2324193375' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28235-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:54.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.840940+0000 mon.b [INF] from='client.? 172.21.15.190:0/4038815624' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi190-28049-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:54.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.841097+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi190-28041-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:54.025 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.841339+0000 mon.b [INF] from='client.? 172.21.15.190:0/686815323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi190-28810-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.841556+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi190-28065-24", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.863034+0000 mon.b [INF] from='client.? 172.21.15.190:0/686815323' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi190-28810-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi190-28810-2"} : dispatch 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.864090+0000 mon.b [INF] from='client.? 172.21.15.190:0/1811424013' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi190-28187-4-cache", "mode": "writeback"} : dispatch 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.870770+0000 mon.b [INF] from='client.? 172.21.15.190:0/3176671009' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28516-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.871176+0000 mon.b [INF] from='client.? 172.21.15.190:0/869891065' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi190-28057-16"} : dispatch 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.877294+0000 mon.b [INF] from='client.? 172.21.15.190:0/4038815624' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi190-28049-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.898399+0000 mon.b [WRN] Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.911053+0000 mon.b [INF] from='client.? 172.21.15.190:0/3666954756' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi190-28915-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.916328+0000 mon.c [INF] from='client.? 172.21.15.190:0/17437294' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.917600+0000 client.admin [INF] threexx 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:52.917957+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-11T05:08:54.026 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.844182+0000 mon.b [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-11T05:08:54.890 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.848182+0000 mon.b [INF] from='client.? 172.21.15.190:0/2837697715' entity='client.admin' api_io: Running main() from gmock_main.cc 2024-09-11T05:08:54.890 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-11T05:08:54.890 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] Global test environment set-up. 2024-09-11T05:08:54.890 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-11T05:08:54.890 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-11T05:08:54.890 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (196 ms) 2024-09-11T05:08:54.890 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (48 ms) 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (9 ms) 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.Checksum (2 ms) 2024-09-11T05:08:54.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (5 ms) 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (3 ms) 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (5 ms) 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.TruncTest (6 ms) 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (12 ms) 2024-09-11T05:08:54.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.RmXattr (12 ms) 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.XattrIter (10 ms) 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] 14 tests from LibRadosIo (318 ms total) 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1225 ms) 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (15 ms) 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (10 ms) 2024-09-11T05:08:54.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (13 ms) 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (23 ms) 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (5 ms) 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (11 ms) 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (17 ms) 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (35 ms) 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (13 ms) 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1367 ms total) 2024-09-11T05:08:54.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: 2024-09-11T05:08:54.903 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] Global test environment tear-down 2024-09-11T05:08:54.903 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9311 ms total) 2024-09-11T05:08:54.903 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ PASSED ] 24 tests. 2024-09-11T05:08:55.905 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: Running main() from gmock_main.cc 2024-09-11T05:08:55.905 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-11T05:08:55.905 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] Global test environment set-up. 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.Stat (261 ms) 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (17 ms) 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.StatNS (34 ms) 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (1 ms) 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] 5 tests from LibRadosStat (314 ms total) 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: 2024-09-11T05:08:55.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1155 ms) 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (27 ms) 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1184 ms total) 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] Global test environment tear-down 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10211 ms total) 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-11T05:08:55.907 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Running main() from gmock_main.cc 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: handle_notify cookie 94338681935440 notify_id 98784247808 notifier_gid 4580 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2839 ms) 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Trying... 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: handle_notify cookie 94338695334576 notify_id 111669149696 notifier_gid 4661 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Waiting for 3.000000000s 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Timed out. 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Flushing... 2024-09-11T05:08:55.908 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Flushed... 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7074 ms) 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (9913 ms total) 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (9913 ms total) 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-11T05:08:55.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: seed 28311 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (275 ms) 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (22 ms) 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (6 ms) 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (30 ms) 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (333 ms total) 2024-09-11T05:08:55.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1167 ms) 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (312 ms) 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (9 ms) 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (16 ms) 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1504 ms total) 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10197 ms total) 2024-09-11T05:08:55.911 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-11T05:08:55.934 INFO:tasks.workunit.client.0.smithi190.stdout: list: Running main() from gmock_main.cc 2024-09-11T05:08:55.934 INFO:tasks.workunit.client.0.smithi190.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-11T05:08:55.935 INFO:tasks.workunit.client.0.smithi190.stdout: list: [----------] Global test environment set-up. 2024-09-11T05:08:55.935 INFO:tasks.workunit.client.0.smithi190.stdout: list: [----------] 3 tests from NeoradosList 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ OK ] NeoradosList.ListObjects (2914 ms) 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3015 ms) 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4068 ms) 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [----------] 3 tests from NeoradosList (9997 ms total) 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [----------] Global test environment tear-down 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [==========] 3 tests from 1 test suite ran. (9997 ms total) 2024-09-11T05:08:55.936 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ PASSED ] 3 tests. 2024-09-11T05:08:57.030 INFO:tasks.workunit.client.0.smithi190.stdout: cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi190-28089-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi190-28089-10"}]': finished 2024-09-11T05:08:57.030 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.848311+0000 mon.b [INF] from='client.? 172.21.15.190:0/1811424013' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi190-28187-4-cache", "mode": "writeback"}]': finished 2024-09-11T05:08:57.030 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.848425+0000 mon.b [INF] from='client.? 172.21.15.190:0/3176671009' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28516-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.848595+0000 mon.b [INF] from='client.? 172.21.15.190:0/869891065' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi190-28057-16"}]': finished 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.848726+0000 mon.b [INF] from='client.? 172.21.15.190:0/4038815624' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi190-28049-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.848934+0000 mon.b [INF] from='client.? 172.21.15.190:0/3666954756' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi190-28915-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.872483+0000 mon.c [INF] from='client.? 172.21.15.190:0/3472994114' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi190-28311-7"} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.873749+0000 mon.c [INF] from='client.? 172.21.15.190:0/869558095' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi190-28065-24"} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.874699+0000 mon.c [INF] from='client.? 172.21.15.190:0/942911270' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi190-28356-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.875640+0000 mon.b [INF] from='client.? 172.21.15.190:0/3666954756' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi190-28915-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi190-28915-2"} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.876514+0000 mon.b [INF] from='client.? 172.21.15.190:0/869891065' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi190-28057-16"} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.877067+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi190-28311-7"} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.877749+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi190-28065-24"} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.891802+0000 mon.b [INF] from='client.? 172.21.15.190:0/3824597767' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi190-28303-7"} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T05:08:53.892324+0000 mon.b [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi190-28356-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T05:08:57.031 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8208 ms) 2024-09-11T05:08:57.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (11290 ms total) 2024-09-11T05:08:57.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: 2024-09-11T05:08:57.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-11T05:08:57.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (11290 ms total) 2024-09-11T05:08:57.036 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-11T05:08:58.344 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: Running main() from gmock_main.cc 2024-09-11T05:08:58.344 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-11T05:08:58.344 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [----------] Global test environment set-up. 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5927 ms) 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6497 ms) 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12424 ms total) 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [----------] Global test environment tear-down 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12424 ms total) 2024-09-11T05:08:58.345 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: seed 28094 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (112 ms) 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (82 ms) 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1172 ms) 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1107 ms) 2024-09-11T05:08:58.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (12 ms) 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (7 ms) 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (10 ms) 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (11 ms) 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2514 ms total) 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1158 ms) 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-11T05:08:58.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (10 ms) 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1063 ms) 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1029 ms) 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (19 ms) 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (16 ms) 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (13 ms) 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (8 ms) 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3317 ms total) 2024-09-11T05:08:58.367 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: 2024-09-11T05:08:58.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-11T05:08:58.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (12748 ms total) 2024-09-11T05:08:58.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: Running main() from gmock_main.cc 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [----------] Global test environment set-up. 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolList (1862 ms) 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2020 ms) 2024-09-11T05:08:59.899 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2021 ms) 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4035 ms) 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1460 ms) 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2533 ms) 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [----------] 6 tests from NeoRadosPools (13931 ms total) 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [----------] Global test environment tear-down 2024-09-11T05:08:59.900 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [==========] 6 tests from 1 test suite ran. (13931 ms total) 2024-09-11T05:08:59.901 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ PASSED ] 6 tests. 2024-09-11T05:08:59.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: Running main() from gmock_main.cc 2024-09-11T05:08:59.906 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] Global test environment set-up. 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (276 ms) 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockShared (192 ms) 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1011 ms) 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1008 ms) 2024-09-11T05:08:59.907 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (17 ms) 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.Unlock (13 ms) 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (12 ms) 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (10 ms) 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] 8 tests from LibRadosLock (2539 ms total) 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-11T05:08:59.908 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1145 ms) 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (7 ms) 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1089 ms) 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1015 ms) 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (15 ms) 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (27 ms) 2024-09-11T05:08:59.909 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-11T05:08:59.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (25 ms) 2024-09-11T05:08:59.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-11T05:08:59.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (8 ms) 2024-09-11T05:08:59.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3331 ms total) 2024-09-11T05:08:59.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: 2024-09-11T05:08:59.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] Global test environment tear-down 2024-09-11T05:08:59.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (14300 ms total) 2024-09-11T05:08:59.910 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-11T05:09:01.829 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1148 ms) 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (18 ms) 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1166 ms total) 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-11T05:09:01.830 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (137 ms) 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3466 ms) 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (225 ms) 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (12 ms) 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 146028888066 notifier_gid 4901 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (20 ms) 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-11T05:09:01.831 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 146028888068 notifier_gid 4901 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (19 ms) 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 146028888067 notifier_gid 4901 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (20 ms) 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 146028888068 notifier_gid 4901 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (20 ms) 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 146028888069 notifier_gid 4901 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (14 ms) 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 146028888070 notifier_gid 4901 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (15 ms) 2024-09-11T05:09:01.832 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-11T05:09:01.833 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: trying... 2024-09-11T05:09:01.833 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 146028888069 notifier_gid 4901 2024-09-11T05:09:01.833 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: timed out 2024-09-11T05:09:01.833 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushing 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_w api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (13 ms) 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (6 ms) 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (18 ms) 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (18 ms) 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (6 ms) 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-11T05:09:02.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (7 ms) 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (12 ms) 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (8 ms) 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (9 ms) 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (44 ms) 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (6171 ms) 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1659 ms) 2024-09-11T05:09:02.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9 ms) 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (9 ms) 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (9215 ms total) 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (16750 ms total) 2024-09-11T05:09:02.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-09-11T05:09:06.339 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-11T05:09:06.339 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-11T05:09:06.339 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-11T05:09:06.339 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-11T05:09:06.339 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-11T05:09:06.339 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify_test_cb 2024-09-11T05:09:06.339 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (311 ms) 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93960402035328 err -107 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1028 ms) 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93960402035328 err -107 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1014 ms) 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4469 notify_id 107374182400 cookie 93960402047168 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (19 ms) 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-11T05:09:06.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4469 notify_id 107374182401 cookie 93960402047168 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (9 ms) 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4469 notify_id 107374182401 cookie 93960402083088 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (11 ms) 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4469 notify_id 107374182401 cookie 93960402066256 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4469 notify_id 107374182401 cookie 93960402070720 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (19 ms) 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4469 notify_id 107374182402 cookie 93960402070720 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4469 notify_id 111669149699 cookie 93960402070720 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3010 ms) 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-11T05:09:06.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93960402070720 err -107 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4469 notify_id 141733920770 cookie 93960402070720 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5027 ms) 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93960402070720 err -107 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1013 ms) 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11461 ms total) 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-11T05:09:06.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-11T05:09:06.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify_test_cb 2024-09-11T05:09:06.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1134 ms) 2024-09-11T05:09:06.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1134 ms total) 2024-09-11T05:09:06.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: 2024-09-11T05:09:06.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-11T05:09:06.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (20623 ms total) 2024-09-11T05:09:06.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: Running main() from gmock_main.cc 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [----------] Global test environment set-up. 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6901 ms) 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (7024 ms) 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8452 ms) 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [----------] 3 tests from NeoradosECList (22377 ms total) 2024-09-11T05:09:08.329 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: 2024-09-11T05:09:08.330 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [----------] Global test environment tear-down 2024-09-11T05:09:08.330 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (22377 ms total) 2024-09-11T05:09:08.330 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-11T05:09:11.333 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: Running main() from gmock_main.cc 2024-09-11T05:09:11.333 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [----------] Global test environment set-up. 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2852 ms) 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3025 ms) 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3020 ms) 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2471 ms) 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3541 ms) 2024-09-11T05:09:11.334 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-11T05:09:11.335 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (2433 ms) 2024-09-11T05:09:11.335 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-11T05:09:11.335 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7997 ms) 2024-09-11T05:09:11.335 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (25339 ms total) 2024-09-11T05:09:11.335 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: 2024-09-11T05:09:11.335 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [----------] Global test environment tear-down 2024-09-11T05:09:11.335 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (25339 ms total) 2024-09-11T05:09:11.335 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-11T05:09:12.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-11T05:09:12.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (3048 ms) 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3045 ms) 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-11T05:09:12.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3028 ms) 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3466 ms) 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2975 ms) 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3000 ms) 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (8021 ms) 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (26583 ms total) 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (26583 ms total) 2024-09-11T05:09:12.366 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-11T05:09:13.419 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-11T05:09:13.419 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [----------] Global test environment set-up. 2024-09-11T05:09:13.419 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-11T05:09:13.419 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-11T05:09:13.419 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5088 ms) 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ OK ] LibRadosService.RegisterLate (137 ms) 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: cluster: 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: id: 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: health: HEALTH_WARN 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 13 pool(s) do not have an application enabled 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: services: 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: mon: 3 daemons, quorum b,c (age 39s), out of quorum: a 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: mgr: x(active, since 78s) 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: osd: 6 osds: 6 up (since 70s), 6 in (since 79s) 2024-09-11T05:09:13.420 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: data: 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: pools: 25 pools, 697 pgs 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: usage: 159 MiB used, 540 GiB / 540 GiB avail 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: pgs: 95.265% pgs unknown 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 3.443% pgs not active 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 664 unknown 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 24 creating+peering 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 9 active+clean 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: cluster: 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: id: 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: health: HEALTH_WARN 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 16 pool(s) do not have an application enabled 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: services: 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: mon: 3 daemons, quorum b,c (age 41s), out of quorum: a 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: mgr: x(active, since 80s) 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: osd: 6 osds: 6 up (since 72s), 6 in (since 81s) 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: data: 2024-09-11T05:09:13.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: pools: 27 pools, 713 pgs 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: objects: 54 objects, 581 KiB 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: usage: 174 MiB used, 540 GiB / 540 GiB avail 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: pgs: 45.302% pgs unknown 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 14.867% pgs not active 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 323 unknown 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 284 active+clean 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 106 creating+peering 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: io: 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: client: 2.7 KiB/s rd, 23 KiB/s wr, 204 op/s rd, 239 op/s wr 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2348 ms) 2024-09-11T05:09:13.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-11T05:09:13.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ OK ] LibRadosService.Status (20069 ms) 2024-09-11T05:09:13.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [----------] 4 tests from LibRadosService (27642 ms total) 2024-09-11T05:09:13.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T05:09:13.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [----------] Global test environment tear-down 2024-09-11T05:09:13.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27642 ms total) 2024-09-11T05:09:13.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ PASSED ] 4 tests. 2024-09-11T05:09:14.373 INFO:tasks.workunit.client.0.smithi190.stdout:atch_notify_pp: flushed 2024-09-11T05:09:14.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3010 ms) 2024-09-11T05:09:14.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: trying... 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 158913789959 notifier_gid 4901 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: timed out 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushing 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushed 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3012 ms) 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: List watches 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify2 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 171798691848 notifier_gid 4901 2024-09-11T05:09:14.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify2 done 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: watch_check 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: unwatch2 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushing 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: done 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3018 ms) 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: List watches 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify2 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94240259217248 notify_id 184683593734 notifier_gid 4901 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify2 done 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: watch_check 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: unwatch2 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushing 2024-09-11T05:09:14.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: done 2024-09-11T05:09:14.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3016 ms) 2024-09-11T05:09:14.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16006 ms total) 2024-09-11T05:09:14.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: 2024-09-11T05:09:14.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-11T05:09:14.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28652 ms total) 2024-09-11T05:09:14.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-11T05:09:16.239 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-11T05:09:16.239 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-11T05:09:16.239 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-11T05:09:16.239 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-11T05:09:16.239 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5077 ms) 2024-09-11T05:09:16.239 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-11T05:09:16.239 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (119 ms) 2024-09-11T05:09:16.239 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20076 ms) 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: attempt 0 of 20 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5165 ms) 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30437 ms total) 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30437 ms total) 2024-09-11T05:09:16.240 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-11T05:09:18.399 INFO:tasks.workunit.client.0.smithi190.stdout: misc: Running main() from gmock_main.cc 2024-09-11T05:09:18.399 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-11T05:09:18.399 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [----------] Global test environment set-up. 2024-09-11T05:09:18.399 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-11T05:09:18.399 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-11T05:09:18.399 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.Version (1879 ms) 2024-09-11T05:09:18.399 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2002 ms) 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.LongName (3027 ms) 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3012 ms) 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2484 ms) 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (2996 ms) 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.Exec (2978 ms) 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-11T05:09:18.400 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2993 ms) 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3042 ms) 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2986 ms) 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2020 ms) 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3014 ms) 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [----------] 12 tests from NeoRadosMisc (32434 ms total) 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [----------] Global test environment tear-down 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [==========] 12 tests from 1 test suite ran. (32434 ms total) 2024-09-11T05:09:18.401 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ PASSED ] 12 tests. 2024-09-11T05:09:22.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: Running main() from gmock_main.cc 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [----------] Global test environment set-up. 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2166 ms) 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3023 ms) 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3026 ms) 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-11T05:09:22.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2459 ms) 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3558 ms) 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4444 ms) 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5004 ms) 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5021 ms) 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7976 ms) 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [----------] 9 tests from LibRadosPools (36677 ms total) 2024-09-11T05:09:22.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: 2024-09-11T05:09:22.348 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [----------] Global test environment tear-down 2024-09-11T05:09:22.348 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (36677 ms total) 2024-09-11T05:09:22.348 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-11T05:09:27.372 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: Running main() from gmock_main.cc 2024-09-11T05:09:27.372 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-11T05:09:27.372 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [----------] Global test environment set-up. 2024-09-11T05:09:27.372 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2864 ms) 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3035 ms) 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3025 ms) 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2457 ms) 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3550 ms) 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (2440 ms) 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-11T05:09:27.373 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3473 ms) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2525 ms) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3020 ms) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3025 ms) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3037 ms) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2911 ms) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3041 ms) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2982 ms) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41385 ms total) 2024-09-11T05:09:27.374 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: 2024-09-11T05:09:27.375 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [----------] Global test environment tear-down 2024-09-11T05:09:27.375 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41385 ms total) 2024-09-11T05:09:27.375 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-11T05:09:27.394 INFO:tasks.workunit.client.0.smithi190.stdout: io: Running main() from gmock_main.cc 2024-09-11T05:09:27.394 INFO:tasks.workunit.client.0.smithi190.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-11T05:09:27.394 INFO:tasks.workunit.client.0.smithi190.stdout: io: [----------] Global test environment set-up. 2024-09-11T05:09:27.395 INFO:tasks.workunit.client.0.smithi190.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-11T05:09:27.395 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-11T05:09:27.395 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.Limits (2929 ms) 2024-09-11T05:09:27.395 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-11T05:09:27.395 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3016 ms) 2024-09-11T05:09:27.395 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-11T05:09:27.396 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.ReadOp (3041 ms) 2024-09-11T05:09:27.396 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-11T05:09:27.396 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.SparseRead (2467 ms) 2024-09-11T05:09:27.396 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-11T05:09:27.396 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3539 ms) 2024-09-11T05:09:27.396 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-11T05:09:27.396 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (2428 ms) 2024-09-11T05:09:27.397 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-11T05:09:27.397 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (2991 ms) 2024-09-11T05:09:27.397 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-11T05:09:27.397 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3022 ms) 2024-09-11T05:09:27.397 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-11T05:09:27.397 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3000 ms) 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.Trunc (3030 ms) 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.Remove (3019 ms) 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2927 ms) 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.RmXattr (3044 ms) 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3007 ms) 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [----------] 14 tests from NeoRadosIo (41460 ms total) 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [----------] Global test environment tear-down 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [==========] 14 tests from 1 test suite ran. (41460 ms total) 2024-09-11T05:09:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ PASSED ] 14 tests. 2024-09-11T05:09:41.361 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2022 ms) 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2006 ms) 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2035 ms) 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1999 ms) 2024-09-11T05:09:41.362 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8063 ms total) 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4059 ms) 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4077 ms) 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4917 ms) 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13053 ms total) 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3022 ms) 2024-09-11T05:09:41.363 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2107 ms) 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1830 ms) 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2029 ms) 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8988 ms total) 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4048 ms) 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4035 ms) 2024-09-11T05:09:41.364 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8083 ms total) 2024-09-11T05:09:41.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: 2024-09-11T05:09:41.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-11T05:09:41.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (55689 ms total) 2024-09-11T05:09:41.365 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-11T05:09:42.934 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T05:09:42.934 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T05:09:42.934 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T05:09:42.937 INFO:tasks.ceph.mon.a:Started 2024-09-11T05:09:42.937 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:09:42.937 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:09:43.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient: get_monmap_and_config 2024-09-11T05:09:43.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient: build_initial_monmap 2024-09-11T05:09:43.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient: monmap: 2024-09-11T05:09:43.069 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:43.069 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:43.068126+0000 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:43.068126+0000 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 1 Processor -- start 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 1 -- start start 2024-09-11T05:09:43.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient: init 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecd40a5b40 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 0x7fecd413db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient(hunting): picked mon.noname-a con 0x7fecd40a7130 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4140090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient(hunting): picked mon.noname-b con 0x7fecd409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd409d490 con 0x7fecd40a5b40 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fecd409e110 con 0x7fecd40a7130 2024-09-11T05:09:43.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fecd4070580 con 0x7fecd409d840 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.064+0000 7fecda187640 10 monclient(hunting): authenticate will time out at 1066.503880s 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 0x7fecd413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 0x7fecd413db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48046/0 (socket says 172.21.15.138:48046) 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 1 -- 172.21.15.138:0/3220259256 learned_addr learned my addr 172.21.15.138:0/3220259256 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient(hunting): get_auth_request con 0x7fecd409d840 auth_method 0 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:43.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 10 monclient(hunting): get_auth_request con 0x7fecd40a7130 auth_method 0 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 1 -- 172.21.15.138:0/3220259256 >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 msgr2=0x7fecd413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 1 -- 172.21.15.138:0/3220259256 >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 msgr2=0x7fecd413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 1 --2- 172.21.15.138:0/3220259256 >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 0x7fecd413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd2ffd640 1 --2- 172.21.15.138:0/3220259256 >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 0x7fecd413db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient(hunting): handle_auth_done global_id 21890 payload 274 2024-09-11T05:09:43.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 1 -- 172.21.15.138:0/3220259256 >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 msgr2=0x7fecd413db40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:09:43.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 1 --2- 172.21.15.138:0/3220259256 >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 0x7fecd413db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 1 -- 172.21.15.138:0/3220259256 >> v1:172.21.15.190:6789/0 conn(0x7fecd40a5b40 legacy=0x7fecd40a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:43.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient: _finish_hunting 0 2024-09-11T05:09:43.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 1 monclient: found mon.noname-b 2024-09-11T05:09:43.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:43.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:09:43.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.068+0000 7fecd3fff640 1 -- 172.21.15.138:0/3220259256 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd409cb80 con 0x7fecd409d840 2024-09-11T05:09:43.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd3fff640 1 -- 172.21.15.138:0/3220259256 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 msgr2=0x7fecd4140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:43.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd3fff640 1 --2- 172.21.15.138:0/3220259256 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4140090 secure :-1 s=READY pgs=5574 cs=0 l=1 rev1=1 crypto rx=0x7feccc009d90 tx=0x7feccc01c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=825c3612de5670c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:43.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 1 -- 172.21.15.138:0/3220259256 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc0ac070 con 0x7fecd409d840 2024-09-11T05:09:43.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:43.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: dump: 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: _finish_auth 0 2024-09-11T05:09:43.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: _check_auth_tickets 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 1 -- 172.21.15.138:0/3220259256 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feccc0a6680 con 0x7fecd409d840 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 5 monclient: authenticate success, global_id 21890 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 4 set_mon_vals no callback set 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 1 -- 172.21.15.138:0/3220259256 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc0a6a50 con 0x7fecd409d840 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd27fc640 10 monclient: dump: 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:43.082 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient: shutdown 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- 172.21.15.138:0/3220259256 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 msgr2=0x7fecd4140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 --2- 172.21.15.138:0/3220259256 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4140090 secure :-1 s=READY pgs=5574 cs=0 l=1 rev1=1 crypto rx=0x7feccc009d90 tx=0x7feccc01c830 comp rx=0 tx=0).stop 2024-09-11T05:09:43.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- 172.21.15.138:0/3220259256 shutdown_connections 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 --2- 172.21.15.138:0/3220259256 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4140090 unknown :-1 s=CLOSED pgs=5574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 --2- 172.21.15.138:0/3220259256 >> v2:172.21.15.138:3300/0 conn(0x7fecd40a7130 0x7fecd413db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- 172.21.15.138:0/3220259256 >> 172.21.15.138:0/3220259256 conn(0x7fecd40aba10 msgr2=0x7fecd40a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- 172.21.15.138:0/3220259256 shutdown_connections 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- 172.21.15.138:0/3220259256 wait complete. 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient: build_initial_monmap 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient: monmap: 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:43.084 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 librados: starting msgr at 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 librados: starting objecter 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 Processor -- start 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- start start 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 librados: setting wanted keys 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 librados: calling monclient init 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient: init 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:09:43.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4121a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient(hunting): picked mon.noname-c con 0x7fecd409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 0x7fecd4138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient(hunting): picked mon.noname-a con 0x7fecd40a5b40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd40a7130 0x7fecd413da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient(hunting): picked mon.noname-b con 0x7fecd40a7130 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fecd4143180 con 0x7fecd40a5b40 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fecd4143000 con 0x7fecd40a7130 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fecd4143300 con 0x7fecd409d840 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:43.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecda187640 10 monclient(hunting): authenticate will time out at 1066.511109s 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 0x7fecd4138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4121a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 0x7fecd4138880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48054/0 (socket says 172.21.15.138:48054) 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 1 -- 172.21.15.138:0/1367035296 learned_addr learned my addr 172.21.15.138:0/1367035296 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 10 monclient(hunting): get_auth_request con 0x7fecd40a5b40 auth_method 0 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient(hunting): get_auth_request con 0x7fecd409d840 auth_method 0 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:43.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 1 -- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 msgr2=0x7fecd4138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 1 -- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 msgr2=0x7fecd4138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 0x7fecd4138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd2ffd640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 0x7fecd4138880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd3fff640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd40a7130 0x7fecd413da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient(hunting): handle_auth_done global_id 21893 payload 824 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 1 -- 172.21.15.138:0/1367035296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd40a7130 msgr2=0x7fecd413da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:43.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd40a7130 0x7fecd413da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 1 -- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 msgr2=0x7fecd4138880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 0x7fecd4138880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 1 monclient: found mon.noname-c 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd37fe640 1 -- 172.21.15.138:0/1367035296 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd4122f30 con 0x7fecd409d840 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd3fff640 10 monclient: get_auth_request con 0x7fecd40a7130 auth_method 0 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.072+0000 7fecd3fff640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd40a7130 0x7fecd413da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecd37fe640 1 -- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 msgr2=0x7fecd4121a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecd37fe640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4121a10 secure :-1 s=READY pgs=5575 cs=0 l=1 rev1=1 crypto rx=0x7fecc800c3e0 tx=0x7fecc800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 client.?.objecter ms_handle_connect 0x7fecd409d840 2024-09-11T05:09:43.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 1 -- 172.21.15.138:0/1367035296 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc80119a0 con 0x7fecd409d840 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: dump: 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:43.090 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: _finish_auth 0 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: _check_auth_tickets 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 1 -- 172.21.15.138:0/1367035296 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc8017070 con 0x7fecd409d840 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 5 monclient: authenticate success, global_id 21893 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 10 monclient: _renew_subs 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd40a2330 con 0x7fecd409d840 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 1 -- 172.21.15.138:0/1367035296 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8011d80 con 0x7fecd409d840 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:43.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 monclient: dump: 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecd1ffb640 4 set_mon_vals no callback set 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 10 client.21893.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 10 monclient: _renew_subs 2024-09-11T05:09:43.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd413f4b0 con 0x7fecd409d840 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 1 librados: init done 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd40a5f20 con 0x7fecd409d840 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 1 -- 172.21.15.138:0/1367035296 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fecc801c460 con 0x7fecd409d840 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 1 --2- 172.21.15.138:0/1367035296 >> v2:172.21.15.190:6800/1562782189 conn(0x7fecc003ced0 0x7fecc003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 1 -- 172.21.15.138:0/1367035296 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(78..78 src has 1..78) ==== 7771+0+0 (secure 0 0 0) 0x7fecc8050190 con 0x7fecd409d840 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 10 client.21893.objecter ms_dispatch 0x7fecd4000ba0 osd_map(78..78 src has 1..78) 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 3 client.21893.objecter handle_osd_map got epochs [78,78] > 0 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 3 client.21893.objecter handle_osd_map decoding full epoch 78 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecbbfff640 20 client.21893.objecter dump_active .. 0 homeless 2024-09-11T05:09:43.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecd2ffd640 1 --2- 172.21.15.138:0/1367035296 >> v2:172.21.15.190:6800/1562782189 conn(0x7fecc003ced0 0x7fecc003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:43.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.076+0000 7fecd2ffd640 10 monclient: get_auth_request con 0x7fecc003ced0 auth_method 0 2024-09-11T05:09:43.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.080+0000 7fecd2ffd640 1 --2- 172.21.15.138:0/1367035296 >> v2:172.21.15.190:6800/1562782189 conn(0x7fecc003ced0 0x7fecc003f390 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7fecd41219a0 tx=0x7fecc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:43.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.080+0000 7fecbbfff640 10 client.21893.objecter ms_handle_connect 0x7fecc003ced0 2024-09-11T05:09:43.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.084+0000 7fecbbfff640 1 -- 172.21.15.138:0/1367035296 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fecc8010cd0 con 0x7fecd409d840 2024-09-11T05:09:43.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.084+0000 7fecbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:43.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.084+0000 7fecbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:43.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.316+0000 7fecda187640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:09:43.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.316+0000 7fecda187640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:43.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.316+0000 7fecda187640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:43.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.316+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fecd4122970 con 0x7fecd409d840 2024-09-11T05:09:43.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecbbfff640 1 -- 172.21.15.138:0/1367035296 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fecc8012420 con 0x7fecd409d840 2024-09-11T05:09:43.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:43.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecbbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:43.321 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:09:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 10 librados: watch_flush enter 2024-09-11T05:09:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 10 librados: watch_flush exit 2024-09-11T05:09:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 20 client.21893.objecter shutdown clearing up homeless session... 2024-09-11T05:09:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 10 client.21893.objecter successfully canceled tick 2024-09-11T05:09:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 >> v2:172.21.15.190:6800/1562782189 conn(0x7fecc003ced0 msgr2=0x7fecc003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 --2- 172.21.15.138:0/1367035296 >> v2:172.21.15.190:6800/1562782189 conn(0x7fecc003ced0 0x7fecc003f390 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7fecd41219a0 tx=0x7fecc4008040 comp rx=0 tx=0).stop 2024-09-11T05:09:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 10 monclient: shutdown 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 msgr2=0x7fecd4121a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4121a10 secure :-1 s=READY pgs=5575 cs=0 l=1 rev1=1 crypto rx=0x7fecc800c3e0 tx=0x7fecc800c8a0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 shutdown_connections 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 --2- 172.21.15.138:0/1367035296 >> v2:172.21.15.190:6800/1562782189 conn(0x7fecc003ced0 0x7fecc003f390 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fecd40a7130 0x7fecd413da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fecd40a5b40 0x7fecd4138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 --2- 172.21.15.138:0/1367035296 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fecd409d840 0x7fecd4121a10 unknown :-1 s=CLOSED pgs=5575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 >> 172.21.15.138:0/1367035296 conn(0x7fecd40aba10 msgr2=0x7fecd40a4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 shutdown_connections 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 -- 172.21.15.138:0/1367035296 wait complete. 2024-09-11T05:09:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:43.320+0000 7fecda187640 1 librados: shutdown 2024-09-11T05:09:46.341 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient: get_monmap_and_config 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient: build_initial_monmap 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient: monmap: 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:46.471356+0000 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:46.471356+0000 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:46.470 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.471 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:46.471 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:46.471 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:46.471 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:46.471 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:46.471 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 1 Processor -- start 2024-09-11T05:09:46.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 1 -- start start 2024-09-11T05:09:46.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient: init 2024-09-11T05:09:46.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:46.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-11T05:09:46.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe69806e0e0 0x7fe69809e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient(hunting): picked mon.noname-c con 0x7fe69806e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:09:46.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe69809e960 0x7fe69809ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient(hunting): picked mon.noname-a con 0x7fe69809e960 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6981356f0 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69806e4c0 con 0x7fe6981356f0 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fe6980a19b0 con 0x7fe69809e960 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe69809f820 con 0x7fe69806e0e0 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe69d440640 10 monclient(hunting): authenticate will time out at 1069.907489s 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe69809e960 0x7fe69809ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe69809e960 0x7fe69809ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48080/0 (socket says 172.21.15.138:48080) 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 1 -- 172.21.15.138:0/4224554503 learned_addr learned my addr 172.21.15.138:0/4224554503 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient(hunting): get_auth_request con 0x7fe69809e960 auth_method 0 2024-09-11T05:09:46.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe696ffd640 1 --2- 172.21.15.138:0/4224554503 >> v2:172.21.15.190:3300/0 conn(0x7fe69806e0e0 0x7fe69809e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient(hunting): handle_auth_done global_id 54105 payload 274 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 1 -- 172.21.15.138:0/4224554503 >> v2:172.21.15.190:3300/0 conn(0x7fe69806e0e0 msgr2=0x7fe69809e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 1 --2- 172.21.15.138:0/4224554503 >> v2:172.21.15.190:3300/0 conn(0x7fe69806e0e0 0x7fe69809e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 1 -- 172.21.15.138:0/4224554503 >> v1:172.21.15.138:6789/0 conn(0x7fe6981356f0 legacy=0x7fe698135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient: _finish_hunting 0 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 1 monclient: found mon.noname-a 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:46.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe6967fc640 1 -- 172.21.15.138:0/4224554503 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69809f540 con 0x7fe69809e960 2024-09-11T05:09:46.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe696ffd640 10 monclient: get_auth_request con 0x7fe69806e0e0 auth_method 0 2024-09-11T05:09:46.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.468+0000 7fe696ffd640 1 --2- 172.21.15.138:0/4224554503 >> v2:172.21.15.190:3300/0 conn(0x7fe69806e0e0 0x7fe69809e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:09:46.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6967fc640 1 -- 172.21.15.138:0/4224554503 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe69809e960 msgr2=0x7fe69809ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:46.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6967fc640 1 --2- 172.21.15.138:0/4224554503 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe69809e960 0x7fe69809ed70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe698069420 tx=0x7fe68801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb09f5ac09c0fa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:46.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 1 -- 172.21.15.138:0/4224554503 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6880a55c0 con 0x7fe69809e960 2024-09-11T05:09:46.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:46.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: dump: 2024-09-11T05:09:46.477 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.477 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: _finish_auth 0 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 1 -- 172.21.15.138:0/4224554503 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6880a5760 con 0x7fe69809e960 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 1 -- 172.21.15.138:0/4224554503 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6880a5b30 con 0x7fe69809e960 2024-09-11T05:09:46.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 5 monclient: authenticate success, global_id 54105 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 4 set_mon_vals no callback set 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe695ffb640 10 monclient: dump: 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.479 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient: shutdown 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- 172.21.15.138:0/4224554503 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe69809e960 msgr2=0x7fe69809ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 --2- 172.21.15.138:0/4224554503 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe69809e960 0x7fe69809ed70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe698069420 tx=0x7fe68801c8b0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- 172.21.15.138:0/4224554503 shutdown_connections 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 --2- 172.21.15.138:0/4224554503 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe69809e960 0x7fe69809ed70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 --2- 172.21.15.138:0/4224554503 >> v2:172.21.15.190:3300/0 conn(0x7fe69806e0e0 0x7fe69809e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- 172.21.15.138:0/4224554503 >> 172.21.15.138:0/4224554503 conn(0x7fe698063640 msgr2=0x7fe698063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- 172.21.15.138:0/4224554503 shutdown_connections 2024-09-11T05:09:46.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- 172.21.15.138:0/4224554503 wait complete. 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient: build_initial_monmap 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient: monmap: 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:46.486 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 librados: starting msgr at 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 librados: starting objecter 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 Processor -- start 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- start start 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 librados: setting wanted keys 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 librados: calling monclient init 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient: init 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe69806e0e0 0x7fe6981cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient(hunting): picked mon.noname-c con 0x7fe69806e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe69809e960 0x7fe698134980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient(hunting): picked mon.noname-b con 0x7fe69809e960 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6981356f0 0x7fe69813a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient(hunting): picked mon.noname-a con 0x7fe6981356f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe69813e2e0 con 0x7fe6981356f0 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe69813e160 con 0x7fe69809e960 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe69813e460 con 0x7fe69806e0e0 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe69d440640 10 monclient(hunting): authenticate will time out at 1069.915038s 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe69806e0e0 0x7fe6981cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe69806e0e0 0x7fe6981cebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:42418/0 (socket says 172.21.15.138:42418) 2024-09-11T05:09:46.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 -- 172.21.15.138:0/697826729 learned_addr learned my addr 172.21.15.138:0/697826729 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient(hunting): get_auth_request con 0x7fe69806e0e0 auth_method 0 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6977fe640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6981356f0 0x7fe69813a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6977fe640 10 monclient(hunting): get_auth_request con 0x7fe6981356f0 auth_method 0 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6967fc640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe69809e960 0x7fe698134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6967fc640 10 monclient(hunting): get_auth_request con 0x7fe69809e960 auth_method 0 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6967fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe6977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient(hunting): handle_auth_done global_id 21914 payload 824 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 -- 172.21.15.138:0/697826729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe69809e960 msgr2=0x7fe698134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe69809e960 0x7fe698134980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 -- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6981356f0 msgr2=0x7fe69813a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6981356f0 0x7fe69813a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 monclient: found mon.noname-c 2024-09-11T05:09:46.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.476+0000 7fe696ffd640 1 -- 172.21.15.138:0/697826729 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69813d550 con 0x7fe69806e0e0 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe696ffd640 1 -- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe69806e0e0 msgr2=0x7fe6981cebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe696ffd640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe69806e0e0 0x7fe6981cebf0 secure :-1 s=READY pgs=5582 cs=0 l=1 rev1=1 crypto rx=0x7fe69000c7f0 tx=0x7fe69000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 client.?.objecter ms_handle_connect 0x7fe69806e0e0 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 1 -- 172.21.15.138:0/697826729 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe690015070 con 0x7fe69806e0e0 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: dump: 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.491 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 1 -- 172.21.15.138:0/697826729 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe690007a30 con 0x7fe69806e0e0 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:46.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 1 -- 172.21.15.138:0/697826729 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe690007e20 con 0x7fe69806e0e0 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 monclient: dump: 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 5 monclient: authenticate success, global_id 21914 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe694ff9640 4 set_mon_vals no callback set 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 10 monclient: _renew_subs 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe69809e110 con 0x7fe69806e0e0 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 10 client.21914.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 10 monclient: _renew_subs 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69811e300 con 0x7fe69806e0e0 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 1 librados: init done 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe69809e420 con 0x7fe69806e0e0 2024-09-11T05:09:46.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 1 -- 172.21.15.138:0/697826729 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fe6900113f0 con 0x7fe69806e0e0 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 1 --2- 172.21.15.138:0/697826729 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe67803cf00 0x7fe67803f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 1 -- 172.21.15.138:0/697826729 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(81..81 src has 1..81) ==== 7878+0+0 (secure 0 0 0) 0x7fe690050730 con 0x7fe69806e0e0 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 10 client.21914.objecter ms_dispatch 0x7fe698000ba0 osd_map(81..81 src has 1..81) 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 3 client.21914.objecter handle_osd_map got epochs [81,81] > 0 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 3 client.21914.objecter handle_osd_map decoding full epoch 81 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe68f7fe640 20 client.21914.objecter dump_active .. 0 homeless 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe6967fc640 1 --2- 172.21.15.138:0/697826729 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe67803cf00 0x7fe67803f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.480+0000 7fe6967fc640 10 monclient: get_auth_request con 0x7fe67803cf00 auth_method 0 2024-09-11T05:09:46.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.484+0000 7fe6967fc640 1 --2- 172.21.15.138:0/697826729 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe67803cf00 0x7fe67803f3c0 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7fe69809f370 tx=0x7fe6880a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:46.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.484+0000 7fe68f7fe640 10 client.21914.objecter ms_handle_connect 0x7fe67803cf00 2024-09-11T05:09:46.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.488+0000 7fe68f7fe640 1 -- 172.21.15.138:0/697826729 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe690021d10 con 0x7fe69806e0e0 2024-09-11T05:09:46.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.488+0000 7fe68f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:46.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.488+0000 7fe68f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:46.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.756+0000 7fe69d440640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:09:46.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.756+0000 7fe69d440640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:46.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.756+0000 7fe69d440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:46.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.756+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe69809e6a0 con 0x7fe69806e0e0 2024-09-11T05:09:46.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.756+0000 7fe68f7fe640 1 -- 172.21.15.138:0/697826729 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fe690018360 con 0x7fe69806e0e0 2024-09-11T05:09:46.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.756+0000 7fe68f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:46.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.756+0000 7fe68f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:46.759 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:09:46.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 10 librados: watch_flush enter 2024-09-11T05:09:46.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 10 librados: watch_flush exit 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 20 client.21914.objecter shutdown clearing up homeless session... 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 10 client.21914.objecter successfully canceled tick 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe67803cf00 msgr2=0x7fe67803f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 --2- 172.21.15.138:0/697826729 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe67803cf00 0x7fe67803f3c0 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 crypto rx=0x7fe69809f370 tx=0x7fe6880a7070 comp rx=0 tx=0).stop 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 10 monclient: shutdown 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe69806e0e0 msgr2=0x7fe6981cebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe69806e0e0 0x7fe6981cebf0 secure :-1 s=READY pgs=5582 cs=0 l=1 rev1=1 crypto rx=0x7fe69000c7f0 tx=0x7fe69000ccb0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 shutdown_connections 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 --2- 172.21.15.138:0/697826729 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe67803cf00 0x7fe67803f3c0 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6981356f0 0x7fe69813a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe69809e960 0x7fe698134980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 --2- 172.21.15.138:0/697826729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe69806e0e0 0x7fe6981cebf0 unknown :-1 s=CLOSED pgs=5582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 >> 172.21.15.138:0/697826729 conn(0x7fe698063640 msgr2=0x7fe6980712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:46.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 shutdown_connections 2024-09-11T05:09:46.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 -- 172.21.15.138:0/697826729 wait complete. 2024-09-11T05:09:46.763 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.760+0000 7fe69d440640 1 librados: shutdown 2024-09-11T05:09:46.778 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:09:46.778 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:09:46.778 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:09:46.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient: get_monmap_and_config 2024-09-11T05:09:46.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient: build_initial_monmap 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient: monmap: 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:46.901900+0000 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:46.901900+0000 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:46.901 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:46.902 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 1 Processor -- start 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 1 -- start start 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient: init 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fc2fc070010 0x7fc2fc135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient(hunting): picked mon.noname-a con 0x7fc2fc070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2fc09e6e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2fc09eb10 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2fc1390e0 con 0x7fc2fc09e6e0 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2fc0a8990 con 0x7fc2fc09eb10 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fc2fc0703f0 con 0x7fc2fc070010 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc300e87640 10 monclient(hunting): authenticate will time out at 1070.337979s 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fc2fc070010 0x7fc2fc135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2f9d9b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc2fc09e6e0 0x7fc2fc1382c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:54834/0 (socket says 172.21.15.138:54834) 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2f9d9b640 1 -- 172.21.15.138:0/3108117902 learned_addr learned my addr 172.21.15.138:0/3108117902 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient(hunting): get_auth_request con 0x7fc2fc070010 auth_method 0 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:46.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:46.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_done global_id 54123 payload 274 2024-09-11T05:09:46.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 1 -- 172.21.15.138:0/3108117902 >> v1:172.21.15.190:6789/0 conn(0x7fc2fc09eb10 legacy=0x7fc2fc1389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:46.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 1 -- 172.21.15.138:0/3108117902 >> v1:172.21.15.138:6789/0 conn(0x7fc2fc09e6e0 legacy=0x7fc2fc1382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:46.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient: _finish_hunting 0 2024-09-11T05:09:46.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 1 monclient: found mon.noname-a 2024-09-11T05:09:46.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:46.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:46.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.900+0000 7fc2fa59c640 1 -- 172.21.15.138:0/3108117902 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc09cbf0 con 0x7fc2fc070010 2024-09-11T05:09:46.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2fa59c640 1 -- 172.21.15.138:0/3108117902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 msgr2=0x7fc2fc135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:46.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2fa59c640 1 --2- 172.21.15.138:0/3108117902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 0x7fc2fc135450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0097c0 tx=0x7fc2ec01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15289a87e4d09aab server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:46.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 1 -- 172.21.15.138:0/3108117902 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec0b0070 con 0x7fc2fc070010 2024-09-11T05:09:46.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: dump: 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: _finish_auth 0 2024-09-11T05:09:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: _check_auth_tickets 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 1 -- 172.21.15.138:0/3108117902 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec0ab600 con 0x7fc2fc070010 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 1 -- 172.21.15.138:0/3108117902 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec0ab9d0 con 0x7fc2fc070010 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc300e87640 5 monclient: authenticate success, global_id 54123 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc300e87640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc300e87640 4 set_mon_vals no callback set 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.904+0000 7fc2f959a640 10 monclient: dump: 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.910 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.910 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.910 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.910 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.910 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.910 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient: shutdown 2024-09-11T05:09:46.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:46.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- 172.21.15.138:0/3108117902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 msgr2=0x7fc2fc135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 --2- 172.21.15.138:0/3108117902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 0x7fc2fc135450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0097c0 tx=0x7fc2ec01c330 comp rx=0 tx=0).stop 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- 172.21.15.138:0/3108117902 shutdown_connections 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 --2- 172.21.15.138:0/3108117902 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 0x7fc2fc135450 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- 172.21.15.138:0/3108117902 >> 172.21.15.138:0/3108117902 conn(0x7fc2fc0ab830 msgr2=0x7fc2fc0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- 172.21.15.138:0/3108117902 shutdown_connections 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- 172.21.15.138:0/3108117902 wait complete. 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient: build_initial_monmap 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient: monmap: 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:46.916 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 librados: starting msgr at 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 librados: starting objecter 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 Processor -- start 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- start start 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 librados: setting wanted keys 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 librados: calling monclient init 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient: init 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:09:46.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 0x7fc2fc137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient(hunting): picked mon.noname-a con 0x7fc2fc070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2fc09e6e0 0x7fc2fc1376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient(hunting): picked mon.noname-b con 0x7fc2fc09e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc2fc09eb10 0x7fc2fc137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient(hunting): picked mon.noname-c con 0x7fc2fc09eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc143e90 con 0x7fc2fc070010 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc144010 con 0x7fc2fc09e6e0 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc2fc144190 con 0x7fc2fc09eb10 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc300e87640 10 monclient(hunting): authenticate will time out at 1070.346399s 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 0x7fc2fc137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 0x7fc2fc137190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48092/0 (socket says 172.21.15.138:48092) 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 -- 172.21.15.138:0/1585786589 learned_addr learned my addr 172.21.15.138:0/1585786589 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient(hunting): get_auth_request con 0x7fc2fc070010 auth_method 0 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc2fc09eb10 0x7fc2fc137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 10 monclient(hunting): get_auth_request con 0x7fc2fc09eb10 auth_method 0 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:46.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_done global_id 54126 payload 824 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc2fc09eb10 msgr2=0x7fc2fc137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc2fc09eb10 0x7fc2fc137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2fc09e6e0 msgr2=0x7fc2fc1376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2fc09e6e0 0x7fc2fc1376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient: _finish_hunting 0 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 monclient: found mon.noname-a 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fa59c640 1 -- 172.21.15.138:0/1585786589 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc130530 con 0x7fc2fc070010 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.908+0000 7fc2fad9d640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc2fc09eb10 0x7fc2fc137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:09:46.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2fa59c640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 msgr2=0x7fc2fc137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2fa59c640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 0x7fc2fc137190 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec01cbd0 tx=0x7fc2ec01ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 client.?.objecter ms_handle_connect 0x7fc2fc070010 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec0b0050 con 0x7fc2fc070010 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: dump: 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.921 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: _finish_auth 0 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: _check_auth_tickets 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec0bc070 con 0x7fc2fc070010 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec0b5c80 con 0x7fc2fc070010 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 5 monclient: authenticate success, global_id 54126 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 10 monclient: _renew_subs 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2ebfff640 4 set_mon_vals no callback set 2024-09-11T05:09:46.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2fc0a2330 con 0x7fc2fc070010 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 monclient: dump: 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.923 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 10 client.54126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 10 monclient: _renew_subs 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2fc130250 con 0x7fc2fc070010 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 1 librados: init done 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0400 0x7fc2fc1d07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc135ba0 con 0x7fc2fc1d0400 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0400 0x7fc2fc1d07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 10 monclient: get_auth_request con 0x7fc2fc1d0400 auth_method 0 2024-09-11T05:09:46.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fc2ec0c1460 con 0x7fc2fc070010 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 10 monclient(hunting): handle_auth_done global_id 54129 payload 824 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2f9d9b640 1 -- 172.21.15.138:0/1585786589 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2fc0aaca0 con 0x7fc2fc1d0400 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 1 --2- 172.21.15.138:0/1585786589 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc2dc03cea0 0x7fc2dc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7095+0+0 (secure 0 0 0) 0x7fc2ec0ea9e0 con 0x7fc2fc070010 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 10 client.54126.objecter ms_dispatch 0x7fc2fc000ba0 osd_map(82..82 src has 1..82) 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 3 client.54126.objecter handle_osd_map got epochs [82,82] > 0 2024-09-11T05:09:46.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 3 client.54126.objecter handle_osd_map decoding full epoch 82 2024-09-11T05:09:46.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2eaffd640 20 client.54126.objecter dump_active .. 0 homeless 2024-09-11T05:09:46.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2fad9d640 1 --2- 172.21.15.138:0/1585786589 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc2dc03cea0 0x7fc2dc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.912+0000 7fc2fad9d640 10 monclient: get_auth_request con 0x7fc2dc03cea0 auth_method 0 2024-09-11T05:09:46.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 10 client.54126.objecter ms_handle_connect 0x7fc2fc1d0400 2024-09-11T05:09:46.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 10 client.54126.objecter resend_mon_ops 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2f9d9b640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0400 msgr2=0x7fc2fc1d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2f9d9b640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0400 0x7fc2fc1d07e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000c310 tx=0x7fc2f0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0017070 con 0x7fc2fc1d0400 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 10 monclient: dump: 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.929 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2f00070d0 con 0x7fc2fc1d0400 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0400 msgr2=0x7fc2fc1d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0400 0x7fc2fc1d07e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000c310 tx=0x7fc2f0005850 comp rx=0 tx=0).stop 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0400 msgr2=0x7fc2fc1d07e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0400 0x7fc2fc1d07e0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000c310 tx=0x7fc2f0005850 comp rx=0 tx=0).stop 2024-09-11T05:09:46.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2eaffd640 10 client.54126.objecter ms_handle_connect 0x7fc2dc03cea0 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.916+0000 7fc2fad9d640 1 --2- 172.21.15.138:0/1585786589 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc2dc03cea0 0x7fc2dc03f360 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc12eda0 tx=0x7fc2f4003d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc300e87640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc300e87640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0d20 0x7fc2fc0a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc300e87640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc135450 con 0x7fc2fc1d0d20 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0d20 0x7fc2fc0a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 10 monclient: get_auth_request con 0x7fc2fc1d0d20 auth_method 0 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 10 monclient(hunting): handle_auth_done global_id 54132 payload 824 2024-09-11T05:09:46.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.920+0000 7fc2f9d9b640 1 -- 172.21.15.138:0/1585786589 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2fc132640 con 0x7fc2fc1d0d20 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2f9d9b640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0d20 msgr2=0x7fc2fc0a2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 10 client.54126.objecter ms_handle_connect 0x7fc2fc1d0d20 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 10 client.54126.objecter resend_mon_ops 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2f9d9b640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0d20 0x7fc2fc0a2500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0014000 tx=0x7fc2f0004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0017030 con 0x7fc2fc1d0d20 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 10 monclient: dump: 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:46.932 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7fc2f0022410 con 0x7fc2fc1d0d20 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0d20 msgr2=0x7fc2fc0a2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0d20 0x7fc2fc0a2500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0014000 tx=0x7fc2f0004950 comp rx=0 tx=0).stop 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0d20 msgr2=0x7fc2fc0a2500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2eaffd640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc1d0d20 0x7fc2fc0a2500 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0014000 tx=0x7fc2f0004950 comp rx=0 tx=0).stop 2024-09-11T05:09:46.933 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 14, 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 3907, 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:09:46.934 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:46.935 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:46.936 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:09:46.937 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:46.938 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:09:46.939 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:46.940 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:46.941 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:46.942 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.943 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.944 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:46.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:46.946 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:09:46.946 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:09:46.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 10 librados: watch_flush enter 2024-09-11T05:09:46.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 10 librados: watch_flush exit 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 20 client.54126.objecter shutdown clearing up homeless session... 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 10 client.54126.objecter successfully canceled tick 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc2dc03cea0 msgr2=0x7fc2dc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 --2- 172.21.15.138:0/1585786589 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc2dc03cea0 0x7fc2dc03f360 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc12eda0 tx=0x7fc2f4003d50 comp rx=0 tx=0).stop 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 10 monclient: shutdown 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 msgr2=0x7fc2fc137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 --2- 172.21.15.138:0/1585786589 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc2fc070010 0x7fc2fc137190 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec01cbd0 tx=0x7fc2ec01ab70 comp rx=0 tx=0).stop 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2fad9d640 1 -- 172.21.15.138:0/1585786589 reap_dead start 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc2fad9d640 1 -- 172.21.15.138:0/1585786589 reap_dead start 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 shutdown_connections 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 >> 172.21.15.138:0/1585786589 conn(0x7fc2fc0ab830 msgr2=0x7fc2fc0a10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:46.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 shutdown_connections 2024-09-11T05:09:46.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 -- 172.21.15.138:0/1585786589 wait complete. 2024-09-11T05:09:46.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:46.924+0000 7fc300e87640 1 librados: shutdown 2024-09-11T05:09:46.949 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:09:46.949 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:09:46.949 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:09:47.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient: get_monmap_and_config 2024-09-11T05:09:47.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient: build_initial_monmap 2024-09-11T05:09:47.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient: monmap: 2024-09-11T05:09:47.064 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:47.064 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.064 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:47.065458+0000 2024-09-11T05:09:47.064 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:47.065458+0000 2024-09-11T05:09:47.065 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:47.065 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.065 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:47.065 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:47.065 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:47.065 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:47.065 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:47.065 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 1 Processor -- start 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 1 -- start start 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient: init 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcab4070010 0x7fcab4135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient(hunting): picked mon.noname-c con 0x7fcab4070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fcab409e6e0 0x7fcab41382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient(hunting): picked mon.noname-a con 0x7fcab409e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcab409eb10 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab40a8990 con 0x7fcab409eb10 2024-09-11T05:09:47.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fcab40703f0 con 0x7fcab409e6e0 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fcab409cbf0 con 0x7fcab4070010 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcabb3d5640 10 monclient(hunting): authenticate will time out at 1070.501525s 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fcab409e6e0 0x7fcab41382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fcab409e6e0 0x7fcab41382c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48114/0 (socket says 172.21.15.138:48114) 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 1 -- 172.21.15.138:0/534506573 learned_addr learned my addr 172.21.15.138:0/534506573 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient(hunting): get_auth_request con 0x7fcab409e6e0 auth_method 0 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab9171640 1 --2- 172.21.15.138:0/534506573 >> v2:172.21.15.190:3300/0 conn(0x7fcab4070010 0x7fcab4135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab9171640 10 monclient(hunting): get_auth_request con 0x7fcab4070010 auth_method 0 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab9171640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab9171640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab9171640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient(hunting): handle_auth_done global_id 54135 payload 274 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 1 -- 172.21.15.138:0/534506573 >> v2:172.21.15.190:3300/0 conn(0x7fcab4070010 msgr2=0x7fcab4135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 1 --2- 172.21.15.138:0/534506573 >> v2:172.21.15.190:3300/0 conn(0x7fcab4070010 0x7fcab4135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 1 -- 172.21.15.138:0/534506573 >> v1:172.21.15.190:6789/0 conn(0x7fcab409eb10 legacy=0x7fcab4138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient: _finish_hunting 0 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 1 monclient: found mon.noname-a 2024-09-11T05:09:47.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:47.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.064+0000 7fcab8970640 1 -- 172.21.15.138:0/534506573 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab409ced0 con 0x7fcab409e6e0 2024-09-11T05:09:47.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcab8970640 1 -- 172.21.15.138:0/534506573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab409e6e0 msgr2=0x7fcab41382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcab8970640 1 --2- 172.21.15.138:0/534506573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab409e6e0 0x7fcab41382c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcab406daa0 tx=0x7fcaac01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e35b11f956526e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 1 -- 172.21.15.138:0/534506573 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0a5720 con 0x7fcab409e6e0 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: dump: 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.073 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: _finish_auth 0 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: _check_auth_tickets 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 1 -- 172.21.15.138:0/534506573 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaac0a58c0 con 0x7fcab409e6e0 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 1 -- 172.21.15.138:0/534506573 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0a5c90 con 0x7fcab409e6e0 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 5 monclient: authenticate success, global_id 54135 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 4 set_mon_vals no callback set 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:47.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcaa7fff640 10 monclient: dump: 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 10 monclient: shutdown 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 1 -- 172.21.15.138:0/534506573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab409e6e0 msgr2=0x7fcab41382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 1 --2- 172.21.15.138:0/534506573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab409e6e0 0x7fcab41382c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcab406daa0 tx=0x7fcaac01c880 comp rx=0 tx=0).stop 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 1 -- 172.21.15.138:0/534506573 shutdown_connections 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 1 --2- 172.21.15.138:0/534506573 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab409e6e0 0x7fcab41382c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 1 --2- 172.21.15.138:0/534506573 >> v2:172.21.15.190:3300/0 conn(0x7fcab4070010 0x7fcab4135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 1 -- 172.21.15.138:0/534506573 >> 172.21.15.138:0/534506573 conn(0x7fcab40ab830 msgr2=0x7fcab40a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 1 -- 172.21.15.138:0/534506573 shutdown_connections 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.068+0000 7fcabb3d5640 1 -- 172.21.15.138:0/534506573 wait complete. 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient: build_initial_monmap 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient: monmap: 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:47.076 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 librados: starting msgr at 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 librados: starting objecter 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 Processor -- start 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 -- start start 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 librados: setting wanted keys 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 librados: calling monclient init 2024-09-11T05:09:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient: init 2024-09-11T05:09:47.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:47.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:09:47.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab4070010 0x7fcab41d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient(hunting): picked mon.noname-a con 0x7fcab4070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab409e6e0 0x7fcab41d0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient(hunting): picked mon.noname-c con 0x7fcab409e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcab409eb10 0x7fcab41d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient(hunting): picked mon.noname-b con 0x7fcab409eb10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4142e80 con 0x7fcab4070010 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4139120 con 0x7fcab409eb10 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcab4143000 con 0x7fcab409e6e0 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcabb3d5640 10 monclient(hunting): authenticate will time out at 1070.509451s 2024-09-11T05:09:47.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab4070010 0x7fcab41d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab4070010 0x7fcab41d02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48122/0 (socket says 172.21.15.138:48122) 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 -- 172.21.15.138:0/3712040470 learned_addr learned my addr 172.21.15.138:0/3712040470 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient(hunting): get_auth_request con 0x7fcab4070010 auth_method 0 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab8970640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab409e6e0 0x7fcab41d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab8970640 10 monclient(hunting): get_auth_request con 0x7fcab409e6e0 auth_method 0 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab8970640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab8970640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab8970640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient(hunting): handle_auth_done global_id 54138 payload 824 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9972640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcab409eb10 0x7fcab41d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab409e6e0 msgr2=0x7fcab41d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab409e6e0 0x7fcab41d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcab409eb10 msgr2=0x7fcab41d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcab409eb10 0x7fcab41d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient: _finish_hunting 0 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 monclient: found mon.noname-a 2024-09-11T05:09:47.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9171640 1 -- 172.21.15.138:0/3712040470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab40a1c10 con 0x7fcab4070010 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9972640 10 monclient: get_auth_request con 0x7fcab409eb10 auth_method 0 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.072+0000 7fcab9972640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcab409eb10 0x7fcab41d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab9171640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab4070010 msgr2=0x7fcab41d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab9171640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab4070010 0x7fcab41d02d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcab000d620 tx=0x7fcab000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 client.?.objecter ms_handle_connect 0x7fcab4070010 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab000b830 con 0x7fcab4070010 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: dump: 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.086 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: _finish_auth 0 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 5 monclient: authenticate success, global_id 54138 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab0019070 con 0x7fcab4070010 2024-09-11T05:09:47.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 10 monclient: _renew_subs 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab40a2ae0 con 0x7fcab4070010 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab000bc30 con 0x7fcab4070010 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 monclient: dump: 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.088 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 10 client.54138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa6ffd640 4 set_mon_vals no callback set 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 10 monclient: _renew_subs 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab40a2330 con 0x7fcab4070010 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 1 librados: init done 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412e860 0x7fcab412ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcab40a3c80 con 0x7fcab412e860 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412e860 0x7fcab412ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 10 monclient: get_auth_request con 0x7fcab412e860 auth_method 0 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fcab0013070 con 0x7fcab4070010 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 10 monclient(hunting): handle_auth_done global_id 21938 payload 824 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 1 -- 172.21.15.138:0/3712040470 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcab4135450 con 0x7fcab412e860 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 1 --2- 172.21.15.138:0/3712040470 >> v2:172.21.15.190:6800/1562782189 conn(0x7fca9803cea0 0x7fca9803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7095+0+0 (secure 0 0 0) 0x7fcab00501b0 con 0x7fcab4070010 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 client.54138.objecter ms_dispatch 0x7fcab4000ba0 osd_map(82..82 src has 1..82) 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 3 client.54138.objecter handle_osd_map got epochs [82,82] > 0 2024-09-11T05:09:47.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 3 client.54138.objecter handle_osd_map decoding full epoch 82 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 20 client.54138.objecter dump_active .. 0 homeless 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab9972640 1 --2- 172.21.15.138:0/3712040470 >> v2:172.21.15.190:6800/1562782189 conn(0x7fca9803cea0 0x7fca9803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab9972640 10 monclient: get_auth_request con 0x7fca9803cea0 auth_method 0 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412e860 msgr2=0x7fcab412ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 client.54138.objecter ms_handle_connect 0x7fcab412e860 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcaa5ffb640 10 client.54138.objecter resend_mon_ops 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.076+0000 7fcab8970640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412e860 0x7fcab412ec40 secure :-1 s=READY pgs=5584 cs=0 l=1 rev1=1 crypto rx=0x7fcab4135200 tx=0x7fcaac019620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0b6800 con 0x7fcab412e860 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 10 monclient: dump: 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.091 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcaac0b6800 con 0x7fcab412e860 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412e860 msgr2=0x7fcab412ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412e860 0x7fcab412ec40 secure :-1 s=READY pgs=5584 cs=0 l=1 rev1=1 crypto rx=0x7fcab4135200 tx=0x7fcaac019620 comp rx=0 tx=0).stop 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412e860 msgr2=0x7fcab412ec40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:47.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412e860 0x7fcab412ec40 secure :-1 s=CLOSED pgs=5584 cs=0 l=1 rev1=1 crypto rx=0x7fcab4135200 tx=0x7fcaac019620 comp rx=0 tx=0).stop 2024-09-11T05:09:47.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcab9972640 1 --2- 172.21.15.138:0/3712040470 >> v2:172.21.15.190:6800/1562782189 conn(0x7fca9803cea0 0x7fca9803f360 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fcab4133850 tx=0x7fcaa8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.080+0000 7fcaa5ffb640 10 client.54138.objecter ms_handle_connect 0x7fca9803cea0 2024-09-11T05:09:47.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcabb3d5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:09:47.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcabb3d5640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412f180 0x7fcab412fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcabb3d5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcab40a0170 con 0x7fcab412f180 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412f180 0x7fcab412fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 10 monclient: get_auth_request con 0x7fcab412f180 auth_method 0 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 10 monclient(hunting): handle_auth_done global_id 21941 payload 824 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.084+0000 7fcab8970640 1 -- 172.21.15.138:0/3712040470 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcab412f590 con 0x7fcab412f180 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcab8970640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412f180 msgr2=0x7fcab412fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 10 client.54138.objecter ms_handle_connect 0x7fcab412f180 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 10 client.54138.objecter resend_mon_ops 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcab8970640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412f180 0x7fcab412fa80 secure :-1 s=READY pgs=5585 cs=0 l=1 rev1=1 crypto rx=0x7fcaac01d120 tx=0x7fcaac0a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac0c5040 con 0x7fcab412f180 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 10 monclient: dump: 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.096 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fcaac0c6a20 con 0x7fcab412f180 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412f180 msgr2=0x7fcab412fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412f180 0x7fcab412fa80 secure :-1 s=READY pgs=5585 cs=0 l=1 rev1=1 crypto rx=0x7fcaac01d120 tx=0x7fcaac0a7000 comp rx=0 tx=0).stop 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412f180 msgr2=0x7fcab412fa80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcaa5ffb640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcab412f180 0x7fcab412fa80 secure :-1 s=CLOSED pgs=5585 cs=0 l=1 rev1=1 crypto rx=0x7fcaac01d120 tx=0x7fcaac0a7000 comp rx=0 tx=0).stop 2024-09-11T05:09:47.097 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 14, 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 145860, 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:09:47.098 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.099 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:47.100 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:09:47.101 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:47.102 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:47.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:09:47.104 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:09:47.105 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:47.106 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.109 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:09:47.110 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:09:47.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 10 librados: watch_flush enter 2024-09-11T05:09:47.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 10 librados: watch_flush exit 2024-09-11T05:09:47.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 20 client.54138.objecter shutdown clearing up homeless session... 2024-09-11T05:09:47.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 10 client.54138.objecter successfully canceled tick 2024-09-11T05:09:47.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 >> v2:172.21.15.190:6800/1562782189 conn(0x7fca9803cea0 msgr2=0x7fca9803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 --2- 172.21.15.138:0/3712040470 >> v2:172.21.15.190:6800/1562782189 conn(0x7fca9803cea0 0x7fca9803f360 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fcab4133850 tx=0x7fcaa8009040 comp rx=0 tx=0).stop 2024-09-11T05:09:47.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 10 monclient: shutdown 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab4070010 msgr2=0x7fcab41d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 --2- 172.21.15.138:0/3712040470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcab4070010 0x7fcab41d02d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcab000d620 tx=0x7fcab000dae0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcab9972640 1 -- 172.21.15.138:0/3712040470 reap_dead start 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcab9972640 1 -- 172.21.15.138:0/3712040470 reap_dead start 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 shutdown_connections 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 >> 172.21.15.138:0/3712040470 conn(0x7fcab40ab830 msgr2=0x7fcab406ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 shutdown_connections 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 -- 172.21.15.138:0/3712040470 wait complete. 2024-09-11T05:09:47.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.088+0000 7fcabb3d5640 1 librados: shutdown 2024-09-11T05:09:47.113 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:09:47.114 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:09:47.114 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:09:47.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient: get_monmap_and_config 2024-09-11T05:09:47.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient: build_initial_monmap 2024-09-11T05:09:47.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient: monmap: 2024-09-11T05:09:47.225 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:47.226835+0000 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:47.226835+0000 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:47.226 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 1 Processor -- start 2024-09-11T05:09:47.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 1 -- start start 2024-09-11T05:09:47.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient: init 2024-09-11T05:09:47.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:47.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f78800a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient(hunting): picked mon.noname-b con 0x7f788009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78800a1e60 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78800a22a0 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7880064590 con 0x7f78800a1e60 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78800a2c70 con 0x7f78800a22a0 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f788009de00 con 0x7f788009da20 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f7886060640 10 monclient(hunting): authenticate will time out at 1070.662912s 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f78800a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f78800a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:42470/0 (socket says 172.21.15.138:42470) 2024-09-11T05:09:47.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 1 -- 172.21.15.138:0/2498070045 learned_addr learned my addr 172.21.15.138:0/2498070045 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient(hunting): get_auth_request con 0x7f788009da20 auth_method 0 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient(hunting): handle_auth_done global_id 21944 payload 274 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 1 -- 172.21.15.138:0/2498070045 >> v1:172.21.15.190:6789/0 conn(0x7f78800a22a0 legacy=0x7f7880136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 1 -- 172.21.15.138:0/2498070045 >> v1:172.21.15.138:6789/0 conn(0x7f78800a1e60 legacy=0x7f7880136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 1 monclient: found mon.noname-b 2024-09-11T05:09:47.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:47.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:09:47.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.224+0000 7f787f7fe640 1 -- 172.21.15.138:0/2498070045 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78800a2f50 con 0x7f788009da20 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787f7fe640 1 -- 172.21.15.138:0/2498070045 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 msgr2=0x7f78800a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787f7fe640 1 --2- 172.21.15.138:0/2498070045 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f78800a1920 secure :-1 s=READY pgs=5586 cs=0 l=1 rev1=1 crypto rx=0x7f7878009e80 tx=0x7f787801c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d78dfb7d76bf585f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 1 -- 172.21.15.138:0/2498070045 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78780a5680 con 0x7f788009da20 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: dump: 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.233 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 1 -- 172.21.15.138:0/2498070045 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78780a5820 con 0x7f788009da20 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f7886060640 5 monclient: authenticate success, global_id 21944 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f7886060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:09:47.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f7886060640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 1 -- 172.21.15.138:0/2498070045 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78780a5bf0 con 0x7f788009da20 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f7886060640 4 set_mon_vals no callback set 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.228+0000 7f787e7fc640 10 monclient: dump: 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.235 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient: shutdown 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- 172.21.15.138:0/2498070045 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 msgr2=0x7f78800a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 --2- 172.21.15.138:0/2498070045 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f78800a1920 secure :-1 s=READY pgs=5586 cs=0 l=1 rev1=1 crypto rx=0x7f7878009e80 tx=0x7f787801c8e0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- 172.21.15.138:0/2498070045 shutdown_connections 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 --2- 172.21.15.138:0/2498070045 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f78800a1920 unknown :-1 s=CLOSED pgs=5586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- 172.21.15.138:0/2498070045 >> 172.21.15.138:0/2498070045 conn(0x7f7880063640 msgr2=0x7f7880063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- 172.21.15.138:0/2498070045 shutdown_connections 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- 172.21.15.138:0/2498070045 wait complete. 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient: build_initial_monmap 2024-09-11T05:09:47.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient: monmap: 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 librados: starting msgr at 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 librados: starting objecter 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 Processor -- start 2024-09-11T05:09:47.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- start start 2024-09-11T05:09:47.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 librados: setting wanted keys 2024-09-11T05:09:47.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 librados: calling monclient init 2024-09-11T05:09:47.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient: init 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f7880073be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient(hunting): picked mon.noname-c con 0x7f788009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78800a1e60 0x7f788009e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient(hunting): picked mon.noname-b con 0x7f78800a1e60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78800a22a0 0x7f788009e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient(hunting): picked mon.noname-a con 0x7f78800a22a0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7880141d70 con 0x7f78800a22a0 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7880141ef0 con 0x7f78800a1e60 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7880142070 con 0x7f788009da20 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f7886060640 10 monclient(hunting): authenticate will time out at 1070.669670s 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f7880073be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f7880073be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:42480/0 (socket says 172.21.15.138:42480) 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 -- 172.21.15.138:0/1994473079 learned_addr learned my addr 172.21.15.138:0/1994473079 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient(hunting): get_auth_request con 0x7f788009da20 auth_method 0 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787ffff640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78800a22a0 0x7f788009e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787ffff640 10 monclient(hunting): get_auth_request con 0x7f78800a22a0 auth_method 0 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient(hunting): handle_auth_done global_id 21947 payload 824 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78800a1e60 msgr2=0x7f788009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78800a1e60 0x7f788009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78800a22a0 msgr2=0x7f788009e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78800a22a0 0x7f788009e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 monclient: found mon.noname-c 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.232+0000 7f787f7fe640 1 -- 172.21.15.138:0/1994473079 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788009fdc0 con 0x7f788009da20 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787f7fe640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 msgr2=0x7f7880073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787f7fe640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f7880073be0 secure :-1 s=READY pgs=5587 cs=0 l=1 rev1=1 crypto rx=0x7f787801c710 tx=0x7f7878019680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 client.?.objecter ms_handle_connect 0x7f788009da20 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78780bc020 con 0x7f788009da20 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: dump: 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.246 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: _finish_auth 0 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: _check_auth_tickets 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:47.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78780b88a0 con 0x7f788009da20 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 5 monclient: authenticate success, global_id 21947 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 10 monclient: _renew_subs 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788009e110 con 0x7f788009da20 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78780b8c90 con 0x7f788009da20 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 10 client.21947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787d7fa640 4 set_mon_vals no callback set 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 monclient: dump: 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.248 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 10 monclient: _renew_subs 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f788013c950 con 0x7f788009da20 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 1 librados: init done 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:47.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788009eea0 0x7f788012d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7880137160 con 0x7f788009eea0 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f78780b6440 con 0x7f788009da20 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 1 --2- 172.21.15.138:0/1994473079 >> v2:172.21.15.190:6800/1562782189 conn(0x7f786403cef0 0x7f786403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788009eea0 0x7f788012d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 10 monclient: get_auth_request con 0x7f788009eea0 auth_method 0 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(82..82 src has 1..82) ==== 7095+0+0 (secure 0 0 0) 0x7f78780eb2f0 con 0x7f788009da20 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 10 client.21947.objecter ms_dispatch 0x7f7880000ba0 osd_map(82..82 src has 1..82) 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 3 client.21947.objecter handle_osd_map got epochs [82,82] > 0 2024-09-11T05:09:47.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 3 client.21947.objecter handle_osd_map decoding full epoch 82 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f7877fff640 20 client.21947.objecter dump_active .. 0 homeless 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787ffff640 1 --2- 172.21.15.138:0/1994473079 >> v2:172.21.15.190:6800/1562782189 conn(0x7f786403cef0 0x7f786403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787ffff640 10 monclient: get_auth_request con 0x7f786403cef0 auth_method 0 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 10 monclient(hunting): handle_auth_done global_id 20023 payload 824 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.236+0000 7f787effd640 1 -- 172.21.15.138:0/1994473079 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f788009f280 con 0x7f788009eea0 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f787effd640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788009eea0 msgr2=0x7f788012d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f787effd640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788009eea0 0x7f788012d880 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 crypto rx=0x7f786800c310 tx=0x7f786800c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 10 client.21947.objecter ms_handle_connect 0x7f788009eea0 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 10 client.21947.objecter resend_mon_ops 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 10 client.21947.objecter ms_handle_connect 0x7f786403cef0 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f787ffff640 1 --2- 172.21.15.138:0/1994473079 >> v2:172.21.15.190:6800/1562782189 conn(0x7f786403cef0 0x7f786403f3b0 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7f7870007fd0 tx=0x7f7870009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868018070 con 0x7f788009eea0 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 10 monclient: dump: 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.252 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7868004800 con 0x7f788009eea0 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788009eea0 msgr2=0x7f788012d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788009eea0 0x7f788012d880 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 crypto rx=0x7f786800c310 tx=0x7f786800c7d0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788009eea0 msgr2=0x7f788012d880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:47.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.240+0000 7f7877fff640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788009eea0 0x7f788012d880 unknown :-1 s=CLOSED pgs=5503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f7886060640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f7886060640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788012ddc0 0x7f788012e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f7886060640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f78800a19a0 con 0x7f788012ddc0 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788012ddc0 0x7f788012e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 10 monclient: get_auth_request con 0x7f788012ddc0 auth_method 0 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 10 monclient(hunting): handle_auth_done global_id 20026 payload 824 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.248+0000 7f787effd640 1 -- 172.21.15.138:0/1994473079 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f788013b9f0 con 0x7f788012ddc0 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 10 client.21947.objecter ms_handle_connect 0x7f788012ddc0 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 10 client.21947.objecter resend_mon_ops 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f787effd640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788012ddc0 msgr2=0x7f788012e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f787effd640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788012ddc0 0x7f788012e1d0 secure :-1 s=READY pgs=5504 cs=0 l=1 rev1=1 crypto rx=0x7f7868015000 tx=0x7f7868003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868018030 con 0x7f788012ddc0 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 10 monclient: dump: 2024-09-11T05:09:47.256 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f786801aa40 con 0x7f788012ddc0 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:09:47.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788012ddc0 msgr2=0x7f788012e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788012ddc0 0x7f788012e1d0 secure :-1 s=READY pgs=5504 cs=0 l=1 rev1=1 crypto rx=0x7f7868015000 tx=0x7f7868003770 comp rx=0 tx=0).stop 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788012ddc0 msgr2=0x7f788012e1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7877fff640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f788012ddc0 0x7f788012e1d0 secure :-1 s=CLOSED pgs=5504 cs=0 l=1 rev1=1 crypto rx=0x7f7868015000 tx=0x7f7868003770 comp rx=0 tx=0).stop 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 14, 2024-09-11T05:09:47.258 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 145985, 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:47.259 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:47.260 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:09:47.261 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:09:47.262 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.263 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:09:47.264 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:47.265 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:09:47.266 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:47.269 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 9 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:47.270 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:47.271 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:09:47.271 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:47.271 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:47.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:47.271 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:09:47.271 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 10 librados: watch_flush enter 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 10 librados: watch_flush exit 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 20 client.21947.objecter shutdown clearing up homeless session... 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 10 client.21947.objecter successfully canceled tick 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 >> v2:172.21.15.190:6800/1562782189 conn(0x7f786403cef0 msgr2=0x7f786403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 --2- 172.21.15.138:0/1994473079 >> v2:172.21.15.190:6800/1562782189 conn(0x7f786403cef0 0x7f786403f3b0 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7f7870007fd0 tx=0x7f7870009040 comp rx=0 tx=0).stop 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 10 monclient: shutdown 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 msgr2=0x7f7880073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 --2- 172.21.15.138:0/1994473079 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f788009da20 0x7f7880073be0 secure :-1 s=READY pgs=5587 cs=0 l=1 rev1=1 crypto rx=0x7f787801c710 tx=0x7f7878019680 comp rx=0 tx=0).stop 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f787ffff640 1 -- 172.21.15.138:0/1994473079 reap_dead start 2024-09-11T05:09:47.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f787ffff640 1 -- 172.21.15.138:0/1994473079 reap_dead start 2024-09-11T05:09:47.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 shutdown_connections 2024-09-11T05:09:47.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 >> 172.21.15.138:0/1994473079 conn(0x7f7880063640 msgr2=0x7f78800a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:47.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 shutdown_connections 2024-09-11T05:09:47.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 -- 172.21.15.138:0/1994473079 wait complete. 2024-09-11T05:09:47.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.252+0000 7f7886060640 1 librados: shutdown 2024-09-11T05:09:47.274 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:09:47.274 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:09:47.274 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.384+0000 7f231c205640 10 monclient: get_monmap_and_config 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.384+0000 7f231c205640 10 monclient: build_initial_monmap 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.384+0000 7f231c205640 10 monclient: monmap: 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:47.388192+0000 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:47.388192+0000 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:47.387 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:47.388 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:47.388 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:47.388 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:47.388 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.384+0000 7f231c205640 1 Processor -- start 2024-09-11T05:09:47.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.384+0000 7f231c205640 1 -- start start 2024-09-11T05:09:47.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.384+0000 7f231c205640 10 monclient: init 2024-09-11T05:09:47.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:47.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T05:09:47.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 0x7f23140a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient(hunting): picked mon.noname-a con 0x7f23140a2fe0 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2314074640 0x7f23140a3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient(hunting): picked mon.noname-c con 0x7f2314074640 addr v2:172.21.15.190:3300/0 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23141387c0 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2314063d40 con 0x7f23141387c0 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f23140a3e40 con 0x7f23140a2fe0 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2314074dd0 con 0x7f2314074640 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231c205640 10 monclient(hunting): authenticate will time out at 1070.824480s 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 0x7f23140a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231a7a2640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f23141387c0 0x7f2314138bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:54854/0 (socket says 172.21.15.138:54854) 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f231a7a2640 1 -- 172.21.15.138:0/4117472643 learned_addr learned my addr 172.21.15.138:0/4117472643 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 10 monclient(hunting): get_auth_request con 0x7f23140a2fe0 auth_method 0 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 1 -- 172.21.15.138:0/4117472643 >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 msgr2=0x7f23140a33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 1 -- 172.21.15.138:0/4117472643 >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 msgr2=0x7f23140a33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 1 --2- 172.21.15.138:0/4117472643 >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 0x7f23140a33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f2319fa1640 1 --2- 172.21.15.138:0/4117472643 >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 0x7f23140a33c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 1 --2- 172.21.15.138:0/4117472643 >> v2:172.21.15.190:3300/0 conn(0x7f2314074640 0x7f23140a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient(hunting): get_auth_request con 0x7f2314074640 auth_method 0 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient(hunting): handle_auth_done global_id 20029 payload 274 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 1 -- 172.21.15.138:0/4117472643 >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 msgr2=0x7f23140a33c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 1 --2- 172.21.15.138:0/4117472643 >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 0x7f23140a33c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 1 -- 172.21.15.138:0/4117472643 >> v1:172.21.15.138:6789/0 conn(0x7f23141387c0 legacy=0x7f2314138bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient: _finish_hunting 0 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 1 monclient: found mon.noname-c 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:09:47.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.388+0000 7f23197a0640 1 -- 172.21.15.138:0/4117472643 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23140a4f80 con 0x7f2314074640 2024-09-11T05:09:47.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f23197a0640 1 -- 172.21.15.138:0/4117472643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2314074640 msgr2=0x7f23140a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f23197a0640 1 --2- 172.21.15.138:0/4117472643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2314074640 0x7f23140a3900 secure :-1 s=READY pgs=5505 cs=0 l=1 rev1=1 crypto rx=0x7f2314069d00 tx=0x7f230801c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59210e422ddb88bf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 1 -- 172.21.15.138:0/4117472643 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23080b0070 con 0x7f2314074640 2024-09-11T05:09:47.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: dump: 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: _finish_auth 0 2024-09-11T05:09:47.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: _check_auth_tickets 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 1 -- 172.21.15.138:0/4117472643 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23080ab600 con 0x7f2314074640 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 5 monclient: authenticate success, global_id 20029 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 4 set_mon_vals no callback set 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 1 -- 172.21.15.138:0/4117472643 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23080a4e60 con 0x7f2314074640 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f2318f9f640 10 monclient: dump: 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.397 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 10 monclient: shutdown 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 -- 172.21.15.138:0/4117472643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2314074640 msgr2=0x7f23140a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 --2- 172.21.15.138:0/4117472643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2314074640 0x7f23140a3900 secure :-1 s=READY pgs=5505 cs=0 l=1 rev1=1 crypto rx=0x7f2314069d00 tx=0x7f230801c300 comp rx=0 tx=0).stop 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 -- 172.21.15.138:0/4117472643 shutdown_connections 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 --2- 172.21.15.138:0/4117472643 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2314074640 0x7f23140a3900 unknown :-1 s=CLOSED pgs=5505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 --2- 172.21.15.138:0/4117472643 >> v2:172.21.15.138:3300/0 conn(0x7f23140a2fe0 0x7f23140a33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 -- 172.21.15.138:0/4117472643 >> 172.21.15.138:0/4117472643 conn(0x7f23140a40d0 msgr2=0x7f23140a44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 -- 172.21.15.138:0/4117472643 shutdown_connections 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 -- 172.21.15.138:0/4117472643 wait complete. 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 10 monclient: build_initial_monmap 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 10 monclient: monmap: 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:47.399 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:47.400 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 librados: starting msgr at 2024-09-11T05:09:47.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 librados: starting objecter 2024-09-11T05:09:47.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 Processor -- start 2024-09-11T05:09:47.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 -- start start 2024-09-11T05:09:47.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 librados: setting wanted keys 2024-09-11T05:09:47.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 1 librados: calling monclient init 2024-09-11T05:09:47.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.392+0000 7f231c205640 10 monclient: init 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2314074640 0x7f231413fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient(hunting): picked mon.noname-c con 0x7f2314074640 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23140a2fe0 0x7f2314137a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient(hunting): picked mon.noname-a con 0x7f23140a2fe0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23141387c0 0x7f2314137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient(hunting): picked mon.noname-b con 0x7f23141387c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2314143640 con 0x7f23140a2fe0 2024-09-11T05:09:47.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f23141434c0 con 0x7f23141387c0 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f23141437c0 con 0x7f2314074640 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient(hunting): authenticate will time out at 1070.832260s 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2314074640 0x7f231413fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23140a2fe0 0x7f2314137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23140a2fe0 0x7f2314137a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48140/0 (socket says 172.21.15.138:48140) 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 -- 172.21.15.138:0/1667379442 learned_addr learned my addr 172.21.15.138:0/1667379442 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient(hunting): get_auth_request con 0x7f23140a2fe0 auth_method 0 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2314074640 0x7f231413fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:42484/0 (socket says 172.21.15.138:42484) 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 10 monclient(hunting): get_auth_request con 0x7f2314074640 auth_method 0 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2319fa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231a7a2640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23141387c0 0x7f2314137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient(hunting): handle_auth_done global_id 54147 payload 824 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 -- 172.21.15.138:0/1667379442 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2314074640 msgr2=0x7f231413fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2314074640 0x7f231413fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 -- 172.21.15.138:0/1667379442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23141387c0 msgr2=0x7f2314137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23141387c0 0x7f2314137f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient: _finish_hunting 0 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 monclient: found mon.noname-a 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 -- 172.21.15.138:0/1667379442 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231409e7c0 con 0x7f23140a2fe0 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231a7a2640 10 monclient: get_auth_request con 0x7f23141387c0 auth_method 0 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231a7a2640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23141387c0 0x7f2314137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 -- 172.21.15.138:0/1667379442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23140a2fe0 msgr2=0x7f2314137a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23197a0640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23140a2fe0 0x7f2314137a50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f23080ae040 tx=0x7f23080190a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 client.?.objecter ms_handle_connect 0x7f23140a2fe0 2024-09-11T05:09:47.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 1 -- 172.21.15.138:0/1667379442 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23080b8020 con 0x7f23140a2fe0 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: dump: 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.408 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: _finish_auth 0 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: _check_auth_tickets 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 5 monclient: authenticate success, global_id 54147 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient: _renew_subs 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f231412ec10 con 0x7f23140a2fe0 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 1 -- 172.21.15.138:0/1667379442 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23080b4620 con 0x7f23140a2fe0 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 1 -- 172.21.15.138:0/1667379442 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23080b0040 con 0x7f23140a2fe0 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f23127fc640 10 monclient: dump: 2024-09-11T05:09:47.409 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 client.54147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient: _renew_subs 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f231412dcb0 con 0x7f23140a2fe0 2024-09-11T05:09:47.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f2313fff640 4 set_mon_vals no callback set 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.396+0000 7f231c205640 1 librados: init done 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f231c205640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f231c205640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f231c205640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f23127fc640 1 -- 172.21.15.138:0/1667379442 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f23080b3050 con 0x7f23140a2fe0 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2314138bd0 con 0x7f23140a2fe0 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f23127fc640 1 --2- 172.21.15.138:0/1667379442 >> v2:172.21.15.190:6800/1562782189 conn(0x7f230003cea0 0x7f230003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f23127fc640 1 -- 172.21.15.138:0/1667379442 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7095+0+0 (secure 0 0 0) 0x7f23080e9ce0 con 0x7f23140a2fe0 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f23127fc640 10 client.54147.objecter ms_dispatch 0x7f2314000ba0 osd_map(82..82 src has 1..82) 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f23127fc640 3 client.54147.objecter handle_osd_map got epochs [82,82] > 0 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f23127fc640 3 client.54147.objecter handle_osd_map decoding full epoch 82 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f23127fc640 20 client.54147.objecter dump_active .. 0 homeless 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f2319fa1640 1 --2- 172.21.15.138:0/1667379442 >> v2:172.21.15.190:6800/1562782189 conn(0x7f230003cea0 0x7f230003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:47.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f2319fa1640 10 monclient: get_auth_request con 0x7f230003cea0 auth_method 0 2024-09-11T05:09:47.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f23127fc640 10 client.54147.objecter ms_handle_connect 0x7f230003cea0 2024-09-11T05:09:47.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.400+0000 7f2319fa1640 1 --2- 172.21.15.138:0/1667379442 >> v2:172.21.15.190:6800/1562782189 conn(0x7f230003cea0 0x7f230003f360 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7f2304009610 tx=0x7f2304009900 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:47.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.404+0000 7f23127fc640 1 -- 172.21.15.138:0/1667379442 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f23080c2030 con 0x7f23140a2fe0 2024-09-11T05:09:47.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.404+0000 7f23127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:47.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.404+0000 7f23127fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:47.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.656+0000 7f231c205640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:09:47.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.656+0000 7f231c205640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:09:47.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.656+0000 7f231c205640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:47.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.656+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f231412cb50 con 0x7f23140a2fe0 2024-09-11T05:09:47.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f23127fc640 1 -- 172.21.15.138:0/1667379442 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f2308016010 con 0x7f23140a2fe0 2024-09-11T05:09:47.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f23127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:09:47.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f23127fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:47.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 10 librados: watch_flush enter 2024-09-11T05:09:47.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 10 librados: watch_flush exit 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 20 client.54147.objecter shutdown clearing up homeless session... 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 10 client.54147.objecter successfully canceled tick 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 >> v2:172.21.15.190:6800/1562782189 conn(0x7f230003cea0 msgr2=0x7f230003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 --2- 172.21.15.138:0/1667379442 >> v2:172.21.15.190:6800/1562782189 conn(0x7f230003cea0 0x7f230003f360 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7f2304009610 tx=0x7f2304009900 comp rx=0 tx=0).stop 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 10 monclient: shutdown 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23140a2fe0 msgr2=0x7f2314137a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23140a2fe0 0x7f2314137a50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f23080ae040 tx=0x7f23080190a0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 shutdown_connections 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 --2- 172.21.15.138:0/1667379442 >> v2:172.21.15.190:6800/1562782189 conn(0x7f230003cea0 0x7f230003f360 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23141387c0 0x7f2314137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23140a2fe0 0x7f2314137a50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 --2- 172.21.15.138:0/1667379442 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2314074640 0x7f231413fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:47.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 >> 172.21.15.138:0/1667379442 conn(0x7f23140a40d0 msgr2=0x7f23140a1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:47.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 shutdown_connections 2024-09-11T05:09:47.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 -- 172.21.15.138:0/1667379442 wait complete. 2024-09-11T05:09:47.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:47.660+0000 7f231c205640 1 librados: shutdown 2024-09-11T05:09:47.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:09:47.880 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:47.876+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:47.880 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:47.876+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3592730597,config=692619927,health=4078015054,kv=310889196,logm=3062991098}) 2024-09-11T05:09:47.880 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:47.876+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=1398305124,config=692619927,health=1839753607,kv=534670036,logm=1622387736}) 2024-09-11T05:09:47.880 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:47.876+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:47.880 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:47.876+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3592730597,config=692619927,health=4078015054,kv=310889196,logm=3062991098}) 2024-09-11T05:09:47.880 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:47.876+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,health=11,kv=10,logm=67} crc {auth=3592730597,health=3738870552,kv=310889196,logm=596019053}) 2024-09-11T05:09:48.025 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.024+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.025 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.024+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2609643129}) 2024-09-11T05:09:48.025 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.024+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374044699}) 2024-09-11T05:09:48.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.024+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.024+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2609643129}) 2024-09-11T05:09:48.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.024+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1997676982}) 2024-09-11T05:09:48.035 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.032+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.035 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.032+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602351847}) 2024-09-11T05:09:48.035 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.032+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=35953058}) 2024-09-11T05:09:48.035 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.032+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.035 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.032+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3602351847}) 2024-09-11T05:09:48.035 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.032+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4128153135}) 2024-09-11T05:09:48.370 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.367+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.370 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.367+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062670513}) 2024-09-11T05:09:48.370 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.367+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511928894}) 2024-09-11T05:09:48.370 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.367+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.370 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.367+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062670513}) 2024-09-11T05:09:48.370 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.367+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284986293}) 2024-09-11T05:09:48.372 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.372 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=908477374}) 2024-09-11T05:09:48.372 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1259387701}) 2024-09-11T05:09:48.372 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.372 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=908477374}) 2024-09-11T05:09:48.373 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348930531}) 2024-09-11T05:09:48.374 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.375 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903985633}) 2024-09-11T05:09:48.375 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846729836}) 2024-09-11T05:09:48.375 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.375 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903985633}) 2024-09-11T05:09:48.375 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.371+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052022809}) 2024-09-11T05:09:48.377 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.377 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057529545}) 2024-09-11T05:09:48.377 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480658851}) 2024-09-11T05:09:48.377 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.377 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057529545}) 2024-09-11T05:09:48.377 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2512230484}) 2024-09-11T05:09:48.379 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.379 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533377806}) 2024-09-11T05:09:48.379 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3048188060}) 2024-09-11T05:09:48.379 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.380 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533377806}) 2024-09-11T05:09:48.380 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.375+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1660818239}) 2024-09-11T05:09:48.381 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.379+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.382 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.379+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988184245}) 2024-09-11T05:09:48.382 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.379+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=986748052}) 2024-09-11T05:09:48.382 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.379+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.382 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.379+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988184245}) 2024-09-11T05:09:48.382 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.379+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2104680395}) 2024-09-11T05:09:48.384 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.379+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.384 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303796853}) 2024-09-11T05:09:48.384 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3291336281}) 2024-09-11T05:09:48.384 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.384 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303796853}) 2024-09-11T05:09:48.384 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2128417388}) 2024-09-11T05:09:48.386 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.386 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760887420}) 2024-09-11T05:09:48.386 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271397920}) 2024-09-11T05:09:48.386 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.386 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760887420}) 2024-09-11T05:09:48.386 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.383+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4281460392}) 2024-09-11T05:09:48.388 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.388 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=88294055,mdsmap=1688852554}) 2024-09-11T05:09:48.389 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3840019261}) 2024-09-11T05:09:48.389 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.389 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=88294055,mdsmap=1688852554}) 2024-09-11T05:09:48.389 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1037291550,mdsmap=1754867906}) 2024-09-11T05:09:48.392 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.392 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=15,osdmap=42} crc {mdsmap=3140514009,mgr=1245745713,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=1578132707,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=881432720,osd_pg_creating=1004311315,osd_snap=2849385098,osdmap=3650441135}) 2024-09-11T05:09:48.392 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=14,osdmap=45} crc {mdsmap=3140514009,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=1578132707,mon_config_key=866245988,monmap=2552663431,nvmeofgw=598337686,osd_metadata=2527622730,osd_pg_creating=1004311315,osd_snap=2562903683,osdmap=4139669412}) 2024-09-11T05:09:48.392 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.387+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.392 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.391+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=15,osdmap=42} crc {mdsmap=3140514009,mgr=1245745713,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=1578132707,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=881432720,osd_pg_creating=1004311315,osd_snap=2849385098,osdmap=3650441135}) 2024-09-11T05:09:48.392 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.391+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=43} crc {mdsmap=3140514009,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=1578132707,mon_config_key=677427989,monmap=967756510,nvmeofgw=1596038039,osd_metadata=3000249921,osd_pg_creating=1004311315,osd_snap=4066492881,osdmap=3176761905}) 2024-09-11T05:09:48.401 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.399+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.401 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.399+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=3579988816}) 2024-09-11T05:09:48.401 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.399+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=95} crc {osdmap=4223629439}) 2024-09-11T05:09:48.401 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.399+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:09:48.401 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.399+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=3579988816}) 2024-09-11T05:09:48.402 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:09:48.399+0000 7ff4a02f8640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=92} crc {osdmap=2045168824}) 2024-09-11T05:09:48.678 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:09:48.679 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient: get_monmap_and_config 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient: build_initial_monmap 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient: monmap: 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:48.806978+0000 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:48.806978+0000 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:48.806 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:48.807 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:48.807 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:48.807 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:48.807 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 1 Processor -- start 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 1 -- start start 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient: init 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5ee806e0e0 0x7f5ee809e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee806e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5ee809e960 0x7f5ee809ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:48.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee809e960 addr v2:172.21.15.190:3300/0 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ee81356f0 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ee806e4c0 con 0x7f5ee81356f0 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f5ee80a19b0 con 0x7f5ee806e0e0 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5ee809f820 con 0x7f5ee809e960 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5ef01ed640 10 monclient(hunting): authenticate will time out at 1072.243568s 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5ee806e0e0 0x7f5ee809e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5ee806e0e0 0x7f5ee809e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:55388/0 (socket says 172.21.15.138:55388) 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 1 -- 172.21.15.138:0/1035377236 learned_addr learned my addr 172.21.15.138:0/1035377236 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 10 monclient(hunting): get_auth_request con 0x7f5ee806e0e0 auth_method 0 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.803+0000 7f5eedf89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eed788640 1 --2- 172.21.15.138:0/1035377236 >> v2:172.21.15.190:3300/0 conn(0x7f5ee809e960 0x7f5ee809ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eed788640 10 monclient(hunting): get_auth_request con 0x7f5ee809e960 auth_method 0 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eed788640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eed788640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eed788640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 10 monclient(hunting): handle_auth_done global_id 54153 payload 274 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 1 -- 172.21.15.138:0/1035377236 >> v2:172.21.15.190:3300/0 conn(0x7f5ee809e960 msgr2=0x7f5ee809ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 1 --2- 172.21.15.138:0/1035377236 >> v2:172.21.15.190:3300/0 conn(0x7f5ee809e960 0x7f5ee809ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 1 -- 172.21.15.138:0/1035377236 >> v1:172.21.15.190:6789/0 conn(0x7f5ee81356f0 legacy=0x7f5ee8135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:48.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 10 monclient: _finish_hunting 0 2024-09-11T05:09:48.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 1 monclient: found mon.noname-a 2024-09-11T05:09:48.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:48.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:48.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.807+0000 7f5eedf89640 1 -- 172.21.15.138:0/1035377236 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee809f540 con 0x7f5ee806e0e0 2024-09-11T05:09:48.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 -- 172.21.15.138:0/1035377236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 msgr2=0x7f5ee809e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:48.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 --2- 172.21.15.138:0/1035377236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 0x7f5ee809e420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80646d0 tx=0x7f5ee401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def37b3a8a74453d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:48.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 1 -- 172.21.15.138:0/1035377236 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40b0070 con 0x7f5ee806e0e0 2024-09-11T05:09:48.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:48.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:48.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: dump: 2024-09-11T05:09:48.820 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:48.820 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:48.821 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:48.821 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:48.821 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:48.821 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:48.821 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: _finish_auth 0 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: _check_auth_tickets 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 5 monclient: authenticate success, global_id 54153 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 1 -- 172.21.15.138:0/1035377236 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee40ab600 con 0x7f5ee806e0e0 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 4 set_mon_vals no callback set 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 1 -- 172.21.15.138:0/1035377236 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40ab9d0 con 0x7f5ee806e0e0 2024-09-11T05:09:48.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eecf87640 10 monclient: dump: 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient: shutdown 2024-09-11T05:09:48.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- 172.21.15.138:0/1035377236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 msgr2=0x7f5ee809e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/1035377236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 0x7f5ee809e420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5ee80646d0 tx=0x7f5ee401c3e0 comp rx=0 tx=0).stop 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- 172.21.15.138:0/1035377236 shutdown_connections 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/1035377236 >> v2:172.21.15.190:3300/0 conn(0x7f5ee809e960 0x7f5ee809ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/1035377236 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 0x7f5ee809e420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- 172.21.15.138:0/1035377236 >> 172.21.15.138:0/1035377236 conn(0x7f5ee8063640 msgr2=0x7f5ee8063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- 172.21.15.138:0/1035377236 shutdown_connections 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- 172.21.15.138:0/1035377236 wait complete. 2024-09-11T05:09:48.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient: build_initial_monmap 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient: monmap: 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:48.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 librados: starting msgr at 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 librados: starting objecter 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 Processor -- start 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- start start 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 librados: setting wanted keys 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 librados: calling monclient init 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient: init 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 0x7f5ee813ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee806e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ee809e960 0x7f5ee813b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee809e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ee81356f0 0x7f5ee813b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:48.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee81356f0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee80a2330 con 0x7f5ee806e0e0 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee80a21b0 con 0x7f5ee81356f0 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee80a24b0 con 0x7f5ee809e960 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5ef01ed640 10 monclient(hunting): authenticate will time out at 1072.250730s 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 0x7f5ee813ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ee809e960 0x7f5ee813b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ee809e960 0x7f5ee813b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59026/0 (socket says 172.21.15.138:59026) 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 1 -- 172.21.15.138:0/907759476 learned_addr learned my addr 172.21.15.138:0/907759476 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 10 monclient(hunting): get_auth_request con 0x7f5ee809e960 auth_method 0 2024-09-11T05:09:48.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:48.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:48.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:48.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient(hunting): get_auth_request con 0x7f5ee806e0e0 auth_method 0 2024-09-11T05:09:48.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:48.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:48.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:48.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:48.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:48.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:48.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:48.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:48.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eed788640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:48.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eee78a640 1 --2- 172.21.15.138:0/907759476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ee81356f0 0x7f5ee813b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:48.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eee78a640 10 monclient(hunting): get_auth_request con 0x7f5ee81356f0 auth_method 0 2024-09-11T05:09:48.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eee78a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:48.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eee78a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:48.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eee78a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:48.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient(hunting): handle_auth_done global_id 54156 payload 824 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 -- 172.21.15.138:0/907759476 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ee809e960 msgr2=0x7f5ee813b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 --2- 172.21.15.138:0/907759476 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ee809e960 0x7f5ee813b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 -- 172.21.15.138:0/907759476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ee81356f0 msgr2=0x7f5ee813b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 --2- 172.21.15.138:0/907759476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ee81356f0 0x7f5ee813b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient: _finish_hunting 0 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 monclient: found mon.noname-a 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.811+0000 7f5eedf89640 1 -- 172.21.15.138:0/907759476 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee809e110 con 0x7f5ee806e0e0 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5eedf89640 1 -- 172.21.15.138:0/907759476 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 msgr2=0x7f5ee813ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5eedf89640 1 --2- 172.21.15.138:0/907759476 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 0x7f5ee813ad50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40ad000 tx=0x7f5ee4019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 client.?.objecter ms_handle_connect 0x7f5ee806e0e0 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:48.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 1 -- 172.21.15.138:0/907759476 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40b0070 con 0x7f5ee806e0e0 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: dump: 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:48.832 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: _finish_auth 0 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 1 -- 172.21.15.138:0/907759476 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee40b1040 con 0x7f5ee806e0e0 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 1 -- 172.21.15.138:0/907759476 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40aa9e0 con 0x7f5ee806e0e0 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 5 monclient: authenticate success, global_id 54156 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 10 monclient: _renew_subs 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee8072b70 con 0x7f5ee806e0e0 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5edffff640 4 set_mon_vals no callback set 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:48.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 monclient: dump: 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 10 client.54156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 10 monclient: _renew_subs 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:48.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee8071850 con 0x7f5ee806e0e0 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 1 librados: init done 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee809e420 con 0x7f5ee806e0e0 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 1 -- 172.21.15.138:0/907759476 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5ee40aabc0 con 0x7f5ee806e0e0 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 1 --2- 172.21.15.138:0/907759476 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5ecc03cef0 0x7f5ecc03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 1 -- 172.21.15.138:0/907759476 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7919+0+0 (secure 0 0 0) 0x7f5ee40ea170 con 0x7f5ee806e0e0 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 10 client.54156.objecter ms_dispatch 0x7f5ee8000ba0 osd_map(84..84 src has 1..84) 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 3 client.54156.objecter handle_osd_map got epochs [84,84] > 0 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 3 client.54156.objecter handle_osd_map decoding full epoch 84 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5ede7fc640 20 client.54156.objecter dump_active .. 0 homeless 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5eed788640 1 --2- 172.21.15.138:0/907759476 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5ecc03cef0 0x7f5ecc03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:48.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.815+0000 7f5eed788640 10 monclient: get_auth_request con 0x7f5ecc03cef0 auth_method 0 2024-09-11T05:09:48.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.819+0000 7f5ede7fc640 10 client.54156.objecter ms_handle_connect 0x7f5ecc03cef0 2024-09-11T05:09:48.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.819+0000 7f5eed788640 1 --2- 172.21.15.138:0/907759476 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5ecc03cef0 0x7f5ecc03f3b0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8074930 tx=0x7f5ed8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:48.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.823+0000 7f5ede7fc640 1 -- 172.21.15.138:0/907759476 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f5ee809e110 con 0x7f5ee806e0e0 2024-09-11T05:09:48.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.823+0000 7f5ede7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:48.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:48.823+0000 7f5ede7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:49.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.059+0000 7f5ef01ed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:09:49.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.059+0000 7f5ef01ed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:49.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.059+0000 7f5ef01ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:49.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.059+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5ee809dea0 con 0x7f5ee806e0e0 2024-09-11T05:09:49.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.059+0000 7f5ede7fc640 1 -- 172.21.15.138:0/907759476 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f5ee40160f0 con 0x7f5ee806e0e0 2024-09-11T05:09:49.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.059+0000 7f5ede7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:49.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.059+0000 7f5ede7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:49.062 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:09:49.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 10 librados: watch_flush enter 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 10 librados: watch_flush exit 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 20 client.54156.objecter shutdown clearing up homeless session... 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 10 client.54156.objecter successfully canceled tick 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5ecc03cef0 msgr2=0x7f5ecc03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/907759476 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5ecc03cef0 0x7f5ecc03f3b0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8074930 tx=0x7f5ed8008040 comp rx=0 tx=0).stop 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 10 monclient: shutdown 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 msgr2=0x7f5ee813ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/907759476 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 0x7f5ee813ad50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40ad000 tx=0x7f5ee4019180 comp rx=0 tx=0).stop 2024-09-11T05:09:49.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 shutdown_connections 2024-09-11T05:09:49.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/907759476 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5ecc03cef0 0x7f5ecc03f3b0 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/907759476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ee81356f0 0x7f5ee813b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/907759476 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ee809e960 0x7f5ee813b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 --2- 172.21.15.138:0/907759476 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ee806e0e0 0x7f5ee813ad50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 >> 172.21.15.138:0/907759476 conn(0x7f5ee8063640 msgr2=0x7f5ee80a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:49.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 shutdown_connections 2024-09-11T05:09:49.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 -- 172.21.15.138:0/907759476 wait complete. 2024-09-11T05:09:49.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.063+0000 7f5ef01ed640 1 librados: shutdown 2024-09-11T05:09:49.081 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:09:49.081 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:09:49.081 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:09:49.081 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:09:49.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient: get_monmap_and_config 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient: build_initial_monmap 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient: monmap: 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:49.198036+0000 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:49.198036+0000 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:49.197 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:49.198 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 1 Processor -- start 2024-09-11T05:09:49.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 1 -- start start 2024-09-11T05:09:49.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient: init 2024-09-11T05:09:49.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f84c00a32c0 0x7f84c009e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient(hunting): picked mon.noname-c con 0x7f84c00a32c0 addr v2:172.21.15.190:3300/0 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84c009e960 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f84c009ed70 0x7f84c013ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient(hunting): picked mon.noname-a con 0x7f84c009ed70 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84c00a1920 con 0x7f84c009e960 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f84c00744a0 con 0x7f84c009ed70 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f84c00a36a0 con 0x7f84c00a32c0 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:49.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c7a61640 10 monclient(hunting): authenticate will time out at 1072.634035s 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c4ffc640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f84c009e960 0x7f84c0139710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:48848/0 (socket says 172.21.15.138:48848) 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f84c009ed70 0x7f84c013ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c4ffc640 1 -- 172.21.15.138:0/1449142111 learned_addr learned my addr 172.21.15.138:0/1449142111 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient(hunting): get_auth_request con 0x7f84c009ed70 auth_method 0 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient(hunting): handle_auth_done global_id 54189 payload 274 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 1 -- 172.21.15.138:0/1449142111 >> v2:172.21.15.190:3300/0 conn(0x7f84c00a32c0 msgr2=0x7f84c009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 1 --2- 172.21.15.138:0/1449142111 >> v2:172.21.15.190:3300/0 conn(0x7f84c00a32c0 0x7f84c009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 1 -- 172.21.15.138:0/1449142111 >> v1:172.21.15.138:6789/0 conn(0x7f84c009e960 legacy=0x7f84c0139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:49.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient: _finish_hunting 0 2024-09-11T05:09:49.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 1 monclient: found mon.noname-a 2024-09-11T05:09:49.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:49.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:49.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.195+0000 7f84c5ffe640 1 -- 172.21.15.138:0/1449142111 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c00644b0 con 0x7f84c009ed70 2024-09-11T05:09:49.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c5ffe640 1 -- 172.21.15.138:0/1449142111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009ed70 msgr2=0x7f84c013ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c5ffe640 1 --2- 172.21.15.138:0/1449142111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009ed70 0x7f84c013ce40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f84c006abd0 tx=0x7f84b801c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282f4f1bd32a689 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 1 -- 172.21.15.138:0/1449142111 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b80ac070 con 0x7f84c009ed70 2024-09-11T05:09:49.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: dump: 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: _finish_auth 0 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: _check_auth_tickets 2024-09-11T05:09:49.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 5 monclient: authenticate success, global_id 54189 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 1 -- 172.21.15.138:0/1449142111 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b80a6700 con 0x7f84c009ed70 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 4 set_mon_vals no callback set 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 1 -- 172.21.15.138:0/1449142111 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b80a6ad0 con 0x7f84c009ed70 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84b7fff640 10 monclient: dump: 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.206 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 10 monclient: shutdown 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 1 -- 172.21.15.138:0/1449142111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009ed70 msgr2=0x7f84c013ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 1 --2- 172.21.15.138:0/1449142111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009ed70 0x7f84c013ce40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f84c006abd0 tx=0x7f84b801c6a0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 1 -- 172.21.15.138:0/1449142111 shutdown_connections 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 1 --2- 172.21.15.138:0/1449142111 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009ed70 0x7f84c013ce40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 1 --2- 172.21.15.138:0/1449142111 >> v2:172.21.15.190:3300/0 conn(0x7f84c00a32c0 0x7f84c009e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 1 -- 172.21.15.138:0/1449142111 >> 172.21.15.138:0/1449142111 conn(0x7f84c0063640 msgr2=0x7f84c0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 1 -- 172.21.15.138:0/1449142111 shutdown_connections 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.199+0000 7f84c7a61640 1 -- 172.21.15.138:0/1449142111 wait complete. 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient: build_initial_monmap 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient: monmap: 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 librados: starting msgr at 2024-09-11T05:09:49.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 librados: starting objecter 2024-09-11T05:09:49.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 Processor -- start 2024-09-11T05:09:49.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 -- start start 2024-09-11T05:09:49.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 librados: setting wanted keys 2024-09-11T05:09:49.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 librados: calling monclient init 2024-09-11T05:09:49.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient: init 2024-09-11T05:09:49.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009e960 0x7f84c013ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient(hunting): picked mon.noname-a con 0x7f84c009e960 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f84c009ed70 0x7f84c01ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient(hunting): picked mon.noname-c con 0x7f84c009ed70 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84c00a32c0 0x7f84c01cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient(hunting): picked mon.noname-b con 0x7f84c00a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f84c013fba0 con 0x7f84c009e960 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f84c009f290 con 0x7f84c00a32c0 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f84c013fd20 con 0x7f84c009ed70 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 10 monclient(hunting): authenticate will time out at 1072.641094s 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f84c009ed70 0x7f84c01ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009e960 0x7f84c013ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009e960 0x7f84c013ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:55410/0 (socket says 172.21.15.138:55410) 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f84c009ed70 0x7f84c01ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59036/0 (socket says 172.21.15.138:59036) 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 -- 172.21.15.138:0/2528307578 learned_addr learned my addr 172.21.15.138:0/2528307578 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient(hunting): get_auth_request con 0x7f84c009e960 auth_method 0 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 10 monclient(hunting): get_auth_request con 0x7f84c009ed70 auth_method 0 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c5ffe640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84c00a32c0 0x7f84c01cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c5ffe640 10 monclient(hunting): get_auth_request con 0x7f84c00a32c0 auth_method 0 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c5ffe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c5ffe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c5ffe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c5ffe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c5ffe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c5ffe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient(hunting): handle_auth_done global_id 54192 payload 824 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f84c009ed70 msgr2=0x7f84c01ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f84c009ed70 0x7f84c01ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84c00a32c0 msgr2=0x7f84c01cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84c00a32c0 0x7f84c01cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient: _finish_hunting 0 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 monclient: found mon.noname-a 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 -- 172.21.15.138:0/2528307578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c01cd940 con 0x7f84c009e960 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009e960 msgr2=0x7f84c013ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c57fd640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009e960 0x7f84c013ad40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00b550 tx=0x7f84bc00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 10 client.?.objecter ms_handle_connect 0x7f84c009e960 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc015020 con 0x7f84c009e960 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 10 monclient: dump: 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.218 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 10 monclient: _finish_auth 0 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84bc011ac0 con 0x7f84c009e960 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.203+0000 7f84c7a61640 5 monclient: authenticate success, global_id 54192 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 10 monclient: _renew_subs 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c0128420 con 0x7f84c009e960 2024-09-11T05:09:49.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0103e0 con 0x7f84c009e960 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: dump: 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.220 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 10 client.54192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 10 monclient: _renew_subs 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b77fe640 4 set_mon_vals no callback set 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c0070d30 con 0x7f84c009e960 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 1 librados: init done 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c013b340 0x7f84c01d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f84c01280a0 con 0x7f84c013b340 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c013b340 0x7f84c01d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 10 monclient: get_auth_request con 0x7f84c013b340 auth_method 0 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f84bc010580 con 0x7f84c009e960 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_done global_id 54195 payload 824 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 1 -- 172.21.15.138:0/2528307578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84c009e420 con 0x7f84c013b340 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 --2- 172.21.15.138:0/2528307578 >> v2:172.21.15.190:6800/1562782189 conn(0x7f84a403ced0 0x7f84a403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7919+0+0 (secure 0 0 0) 0x7f84bc050810 con 0x7f84c009e960 2024-09-11T05:09:49.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 client.54192.objecter ms_dispatch 0x7f84c0000ba0 osd_map(84..84 src has 1..84) 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 3 client.54192.objecter handle_osd_map got epochs [84,84] > 0 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 3 client.54192.objecter handle_osd_map decoding full epoch 84 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 20 client.54192.objecter dump_active .. 0 homeless 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c5ffe640 1 --2- 172.21.15.138:0/2528307578 >> v2:172.21.15.190:6800/1562782189 conn(0x7f84a403ced0 0x7f84a403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c5ffe640 10 monclient: get_auth_request con 0x7f84a403ced0 auth_method 0 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c013b340 msgr2=0x7f84c01d0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 client.54192.objecter ms_handle_connect 0x7f84c013b340 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 client.54192.objecter resend_mon_ops 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84c4ffc640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c013b340 0x7f84c01d0ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84c00714c0 tx=0x7f84b000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0017070 con 0x7f84c013b340 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: dump: 2024-09-11T05:09:49.223 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.224 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f84b000c980 con 0x7f84c013b340 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c013b340 msgr2=0x7f84c01d0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c013b340 0x7f84c01d0ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84c00714c0 tx=0x7f84b000ec00 comp rx=0 tx=0).stop 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c013b340 msgr2=0x7f84c01d0ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.207+0000 7f84b5ffb640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c013b340 0x7f84c01d0ba0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f84c00714c0 tx=0x7f84b000ec00 comp rx=0 tx=0).stop 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.211+0000 7f84b5ffb640 10 client.54192.objecter ms_handle_connect 0x7f84a403ced0 2024-09-11T05:09:49.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.211+0000 7f84c5ffe640 1 --2- 172.21.15.138:0/2528307578 >> v2:172.21.15.190:6800/1562782189 conn(0x7f84a403ced0 0x7f84a403f390 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7f84c006e4c0 tx=0x7f84b80a7000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c7a61640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:09:49.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c7a61640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c002d450 0x7f84c002d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c7a61640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f84c013d070 con 0x7f84c002d450 2024-09-11T05:09:49.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c002d450 0x7f84c002d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 10 monclient: get_auth_request con 0x7f84c002d450 auth_method 0 2024-09-11T05:09:49.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 10 monclient(hunting): handle_auth_done global_id 54198 payload 824 2024-09-11T05:09:49.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.215+0000 7f84c4ffc640 1 -- 172.21.15.138:0/2528307578 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f84c006cca0 con 0x7f84c002d450 2024-09-11T05:09:49.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c4ffc640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c002d450 msgr2=0x7f84c002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 10 client.54192.objecter ms_handle_connect 0x7f84c002d450 2024-09-11T05:09:49.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 10 client.54192.objecter resend_mon_ops 2024-09-11T05:09:49.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c4ffc640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c002d450 0x7f84c002d860 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f84c00714c0 tx=0x7f84b00058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0017070 con 0x7f84c002d450 2024-09-11T05:09:49.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:09:49.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 10 monclient: dump: 2024-09-11T05:09:49.230 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.230 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.230 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.231 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.231 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.231 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.231 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.231 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.231 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.232 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f84b0006600 con 0x7f84c002d450 2024-09-11T05:09:49.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:09:49.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:49.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c002d450 msgr2=0x7f84c002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c002d450 0x7f84c002d860 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f84c00714c0 tx=0x7f84b00058e0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c002d450 msgr2=0x7f84c002d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:49.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84b5ffb640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c002d450 0x7f84c002d860 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f84c00714c0 tx=0x7f84b00058e0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.233 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:09:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:09:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:09:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:09:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 14, 2024-09-11T05:09:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 6203, 2024-09-11T05:09:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:09:49.234 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:09:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:09:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:09:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 4, 2024-09-11T05:09:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:49.235 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:09:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:09:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.236 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.237 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:09:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:09:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.238 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.239 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:09:49.240 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:09:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:09:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:09:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:09:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:09:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:49.241 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:09:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:09:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:09:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:09:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:09:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:09:49.242 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:09:49.243 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:49.243 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:09:49.243 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.243 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.243 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.244 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:09:49.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:09:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:09:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:09:49.246 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:09:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.248 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.248 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:09:49.248 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:09:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:09:49.249 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:09:49.250 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:09:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.251 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:09:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:09:49.252 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:09:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:09:49.253 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.256 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:09:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:09:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.257 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:09:49.258 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.258 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.258 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:09:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:09:49.259 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:09:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:09:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:09:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.261 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:09:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:09:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.262 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:09:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.263 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:09:49.263 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:09:49.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 10 librados: watch_flush enter 2024-09-11T05:09:49.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 10 librados: watch_flush exit 2024-09-11T05:09:49.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 20 client.54192.objecter shutdown clearing up homeless session... 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 10 client.54192.objecter successfully canceled tick 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 >> v2:172.21.15.190:6800/1562782189 conn(0x7f84a403ced0 msgr2=0x7f84a403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 --2- 172.21.15.138:0/2528307578 >> v2:172.21.15.190:6800/1562782189 conn(0x7f84a403ced0 0x7f84a403f390 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7f84c006e4c0 tx=0x7f84b80a7000 comp rx=0 tx=0).stop 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 10 monclient: shutdown 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009e960 msgr2=0x7f84c013ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 --2- 172.21.15.138:0/2528307578 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f84c009e960 0x7f84c013ad40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00b550 tx=0x7f84bc00ba10 comp rx=0 tx=0).stop 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c5ffe640 1 -- 172.21.15.138:0/2528307578 reap_dead start 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c5ffe640 1 -- 172.21.15.138:0/2528307578 reap_dead start 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 shutdown_connections 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 >> 172.21.15.138:0/2528307578 conn(0x7f84c0063640 msgr2=0x7f84c00a2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 shutdown_connections 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 -- 172.21.15.138:0/2528307578 wait complete. 2024-09-11T05:09:49.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.219+0000 7f84c7a61640 1 librados: shutdown 2024-09-11T05:09:49.269 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:09:49.269 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:09:49.269 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:09:49.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.375+0000 7fe769468640 10 monclient: get_monmap_and_config 2024-09-11T05:09:49.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.375+0000 7fe769468640 10 monclient: build_initial_monmap 2024-09-11T05:09:49.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.375+0000 7fe769468640 10 monclient: monmap: 2024-09-11T05:09:49.383 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:49.384 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.384 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:49.380619+0000 2024-09-11T05:09:49.384 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:49.380619+0000 2024-09-11T05:09:49.385 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:49.385 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.385 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:49.385 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:49.385 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:49.385 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:49.386 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:49.386 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.375+0000 7fe769468640 1 Processor -- start 2024-09-11T05:09:49.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.375+0000 7fe769468640 1 -- start start 2024-09-11T05:09:49.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.375+0000 7fe769468640 10 monclient: init 2024-09-11T05:09:49.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:49.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-11T05:09:49.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe764070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:49.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe76409e6e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:49.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe76409eb10 0x7fe764140d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 10 monclient(hunting): picked mon.noname-c con 0x7fe76409eb10 addr v2:172.21.15.190:3300/0 2024-09-11T05:09:49.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe764135b60 con 0x7fe76409e6e0 2024-09-11T05:09:49.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7640a8990 con 0x7fe764070010 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe7640703f0 con 0x7fe76409eb10 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe762ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe764070010 0x7fe764135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.138:55958/0 (socket says 172.21.15.138:55958) 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe762ffd640 1 -- 172.21.15.138:0/3209102169 learned_addr learned my addr 172.21.15.138:0/3209102169 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe769468640 10 monclient(hunting): authenticate will time out at 1072.816514s 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 1 --2- 172.21.15.138:0/3209102169 >> v2:172.21.15.190:3300/0 conn(0x7fe76409eb10 0x7fe764140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient(hunting): get_auth_request con 0x7fe76409eb10 auth_method 0 2024-09-11T05:09:49.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient(hunting): handle_auth_done global_id 20122 payload 274 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 1 -- 172.21.15.138:0/3209102169 >> v1:172.21.15.190:6789/0 conn(0x7fe764070010 legacy=0x7fe764135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 1 -- 172.21.15.138:0/3209102169 >> v1:172.21.15.138:6789/0 conn(0x7fe76409e6e0 legacy=0x7fe76413d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient: _finish_hunting 0 2024-09-11T05:09:49.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 1 monclient: found mon.noname-c 2024-09-11T05:09:49.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:49.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:09:49.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.379+0000 7fe7637fe640 1 -- 172.21.15.138:0/3209102169 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76409cbf0 con 0x7fe76409eb10 2024-09-11T05:09:49.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 1 -- 172.21.15.138:0/3209102169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76409eb10 msgr2=0x7fe764140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 1 --2- 172.21.15.138:0/3209102169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76409eb10 0x7fe764140d70 secure :-1 s=READY pgs=5548 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009d90 tx=0x7fe75c01c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81e02331bcced192 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 1 -- 172.21.15.138:0/3209102169 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0b2070 con 0x7fe76409eb10 2024-09-11T05:09:49.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:49.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: dump: 2024-09-11T05:09:49.401 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.401 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.401 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.401 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.401 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.401 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.402 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.402 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.402 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.402 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: _finish_auth 0 2024-09-11T05:09:49.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:09:49.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:49.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 1 -- 172.21.15.138:0/3209102169 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe75c0a5920 con 0x7fe76409eb10 2024-09-11T05:09:49.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:49.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 1 -- 172.21.15.138:0/3209102169 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0a5cf0 con 0x7fe76409eb10 2024-09-11T05:09:49.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 5 monclient: authenticate success, global_id 20122 2024-09-11T05:09:49.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:49.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 4 set_mon_vals no callback set 2024-09-11T05:09:49.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:09:49.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe761ffb640 10 monclient: dump: 2024-09-11T05:09:49.404 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.404 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.405 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.405 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.405 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.405 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.405 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.405 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.406 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.406 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient: shutdown 2024-09-11T05:09:49.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:49.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- 172.21.15.138:0/3209102169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76409eb10 msgr2=0x7fe764140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 --2- 172.21.15.138:0/3209102169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76409eb10 0x7fe764140d70 secure :-1 s=READY pgs=5548 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009d90 tx=0x7fe75c01c800 comp rx=0 tx=0).stop 2024-09-11T05:09:49.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- 172.21.15.138:0/3209102169 shutdown_connections 2024-09-11T05:09:49.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 --2- 172.21.15.138:0/3209102169 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe76409eb10 0x7fe764140d70 unknown :-1 s=CLOSED pgs=5548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- 172.21.15.138:0/3209102169 >> 172.21.15.138:0/3209102169 conn(0x7fe7640ab830 msgr2=0x7fe7640a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:49.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- 172.21.15.138:0/3209102169 shutdown_connections 2024-09-11T05:09:49.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- 172.21.15.138:0/3209102169 wait complete. 2024-09-11T05:09:49.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient: build_initial_monmap 2024-09-11T05:09:49.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient: monmap: 2024-09-11T05:09:49.408 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:49.408 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:49.408 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:49.408 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:49.409 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:49.409 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.409 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:49.409 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:49.409 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:49.409 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 librados: starting msgr at 2024-09-11T05:09:49.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 librados: starting objecter 2024-09-11T05:09:49.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 Processor -- start 2024-09-11T05:09:49.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- start start 2024-09-11T05:09:49.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 librados: setting wanted keys 2024-09-11T05:09:49.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 librados: calling monclient init 2024-09-11T05:09:49.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient: init 2024-09-11T05:09:49.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:49.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:09:49.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe764070010 0x7fe764131b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient(hunting): picked mon.noname-b con 0x7fe764070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:49.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe76409e6e0 0x7fe7641320c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient(hunting): picked mon.noname-a con 0x7fe76409e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:49.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76409eb10 0x7fe764136e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient(hunting): picked mon.noname-c con 0x7fe76409eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:49.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe764143e60 con 0x7fe76409e6e0 2024-09-11T05:09:49.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe764143ce0 con 0x7fe764070010 2024-09-11T05:09:49.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe764143fe0 con 0x7fe76409eb10 2024-09-11T05:09:49.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:49.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe769468640 10 monclient(hunting): authenticate will time out at 1072.823559s 2024-09-11T05:09:49.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe76409e6e0 0x7fe7641320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76409eb10 0x7fe764136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe762ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe764070010 0x7fe764131b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe76409e6e0 0x7fe7641320c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:55450/0 (socket says 172.21.15.138:55450) 2024-09-11T05:09:49.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 1 -- 172.21.15.138:0/208754747 learned_addr learned my addr 172.21.15.138:0/208754747 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:49.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 10 monclient(hunting): get_auth_request con 0x7fe76409eb10 auth_method 0 2024-09-11T05:09:49.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe762ffd640 10 monclient(hunting): get_auth_request con 0x7fe764070010 auth_method 0 2024-09-11T05:09:49.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe762ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe762ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 10 monclient(hunting): get_auth_request con 0x7fe76409e6e0 auth_method 0 2024-09-11T05:09:49.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe7627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.383+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 10 monclient(hunting): handle_auth_done global_id 22022 payload 824 2024-09-11T05:09:49.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe764070010 msgr2=0x7fe764131b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe764070010 0x7fe764131b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe76409e6e0 msgr2=0x7fe7641320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe76409e6e0 0x7fe7641320c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 10 monclient: _finish_hunting 0 2024-09-11T05:09:49.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 1 monclient: found mon.noname-c 2024-09-11T05:09:49.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:49.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 1 -- 172.21.15.138:0/208754747 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe764133530 con 0x7fe76409eb10 2024-09-11T05:09:49.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe764070010 0x7fe764131b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:09:49.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7627fc640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe76409e6e0 0x7fe7641320c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:09:49.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76409eb10 msgr2=0x7fe764136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7637fe640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76409eb10 0x7fe764136e10 secure :-1 s=READY pgs=5590 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0ac000 tx=0x7fe75c0195a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 client.?.objecter ms_handle_connect 0x7fe76409eb10 2024-09-11T05:09:49.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:49.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0bd030 con 0x7fe76409eb10 2024-09-11T05:09:49.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:49.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: dump: 2024-09-11T05:09:49.422 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.422 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.422 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.422 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.422 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.422 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.423 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.423 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.423 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.423 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:09:49.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:09:49.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:49.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 5 monclient: authenticate success, global_id 22022 2024-09-11T05:09:49.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 10 monclient: _renew_subs 2024-09-11T05:09:49.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe75c0b8680 con 0x7fe76409eb10 2024-09-11T05:09:49.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7641387f0 con 0x7fe76409eb10 2024-09-11T05:09:49.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:49.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0b2040 con 0x7fe76409eb10 2024-09-11T05:09:49.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:49.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 monclient: dump: 2024-09-11T05:09:49.425 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.425 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.425 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.425 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.425 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.425 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.425 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.426 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.426 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.426 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe7617fa640 4 set_mon_vals no callback set 2024-09-11T05:09:49.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 10 client.22022.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:49.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 10 monclient: _renew_subs 2024-09-11T05:09:49.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe764139350 con 0x7fe76409eb10 2024-09-11T05:09:49.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 1 librados: init done 2024-09-11T05:09:49.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76413b4c0 0x7fe764137350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe7640a2390 con 0x7fe76413b4c0 2024-09-11T05:09:49.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76413b4c0 0x7fe764137350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 10 monclient: get_auth_request con 0x7fe76413b4c0 auth_method 0 2024-09-11T05:09:49.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 10 monclient(hunting): handle_auth_done global_id 22025 payload 824 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fe75c0ca060 con 0x7fe76409eb10 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe762ffd640 1 -- 172.21.15.138:0/208754747 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe764135450 con 0x7fe76413b4c0 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 1 --2- 172.21.15.138:0/208754747 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe75003cea0 0x7fe75003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(84..84 src has 1..84) ==== 7919+0+0 (secure 0 0 0) 0x7fe75c0eb300 con 0x7fe76409eb10 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 10 client.22022.objecter ms_dispatch 0x7fe764000ba0 osd_map(84..84 src has 1..84) 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 3 client.22022.objecter handle_osd_map got epochs [84,84] > 0 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.387+0000 7fe74b7fe640 3 client.22022.objecter handle_osd_map decoding full epoch 84 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 20 client.22022.objecter dump_active .. 0 homeless 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe7627fc640 1 --2- 172.21.15.138:0/208754747 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe75003cea0 0x7fe75003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe7627fc640 10 monclient: get_auth_request con 0x7fe75003cea0 auth_method 0 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe762ffd640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76413b4c0 msgr2=0x7fe764137350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe762ffd640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76413b4c0 0x7fe764137350 secure :-1 s=READY pgs=5591 cs=0 l=1 rev1=1 crypto rx=0x7fe7540072a0 tx=0x7fe754007760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 10 client.22022.objecter ms_handle_connect 0x7fe76413b4c0 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 10 client.22022.objecter resend_mon_ops 2024-09-11T05:09:49.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75400fce0 con 0x7fe76413b4c0 2024-09-11T05:09:49.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:49.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 10 monclient: dump: 2024-09-11T05:09:49.430 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.430 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.430 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.430 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.430 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe75400fce0 con 0x7fe76413b4c0 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76413b4c0 msgr2=0x7fe764137350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76413b4c0 0x7fe764137350 secure :-1 s=READY pgs=5591 cs=0 l=1 rev1=1 crypto rx=0x7fe7540072a0 tx=0x7fe754007760 comp rx=0 tx=0).stop 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76413b4c0 msgr2=0x7fe764137350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:49.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76413b4c0 0x7fe764137350 unknown :-1 s=CLOSED pgs=5591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe74b7fe640 10 client.22022.objecter ms_handle_connect 0x7fe75003cea0 2024-09-11T05:09:49.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.391+0000 7fe7627fc640 1 --2- 172.21.15.138:0/208754747 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe75003cea0 0x7fe75003f360 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7fe7640a91a0 tx=0x7fe758003700 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:09:49.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe764137890 0x7fe764137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe7641381b0 con 0x7fe764137890 2024-09-11T05:09:49.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe764137890 0x7fe764137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 10 monclient: get_auth_request con 0x7fe764137890 auth_method 0 2024-09-11T05:09:49.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 10 monclient(hunting): handle_auth_done global_id 22031 payload 824 2024-09-11T05:09:49.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 1 -- 172.21.15.138:0/208754747 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe76413a390 con 0x7fe764137890 2024-09-11T05:09:49.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe764137890 msgr2=0x7fe764137c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe762ffd640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe764137890 0x7fe764137c70 secure :-1 s=READY pgs=5592 cs=0 l=1 rev1=1 crypto rx=0x7fe754017000 tx=0x7fe754004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 10 client.22022.objecter ms_handle_connect 0x7fe764137890 2024-09-11T05:09:49.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 10 client.22022.objecter resend_mon_ops 2024-09-11T05:09:49.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754006450 con 0x7fe764137890 2024-09-11T05:09:49.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:49.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 10 monclient: dump: 2024-09-11T05:09:49.437 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.437 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.437 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.437 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fe754006450 con 0x7fe764137890 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe764137890 msgr2=0x7fe764137c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe764137890 0x7fe764137c70 secure :-1 s=READY pgs=5592 cs=0 l=1 rev1=1 crypto rx=0x7fe754017000 tx=0x7fe754004950 comp rx=0 tx=0).stop 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe764137890 msgr2=0x7fe764137c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:49.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe74b7fe640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe764137890 0x7fe764137c70 secure :-1 s=CLOSED pgs=5592 cs=0 l=1 rev1=1 crypto rx=0x7fe754017000 tx=0x7fe754004950 comp rx=0 tx=0).stop 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 14, 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 148172, 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 4, 2024-09-11T05:09:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:09:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:09:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:09:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:09:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.450 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.451 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.452 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.452 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:09:49.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.452 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.452 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:09:49.452 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:09:49.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 10 librados: watch_flush enter 2024-09-11T05:09:49.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 10 librados: watch_flush exit 2024-09-11T05:09:49.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 20 client.22022.objecter shutdown clearing up homeless session... 2024-09-11T05:09:49.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 10 client.22022.objecter successfully canceled tick 2024-09-11T05:09:49.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe75003cea0 msgr2=0x7fe75003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 1 --2- 172.21.15.138:0/208754747 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe75003cea0 0x7fe75003f360 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7fe7640a91a0 tx=0x7fe758003700 comp rx=0 tx=0).stop 2024-09-11T05:09:49.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 10 monclient: shutdown 2024-09-11T05:09:49.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76409eb10 msgr2=0x7fe764136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe769468640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76409eb10 0x7fe764136e10 secure :-1 s=READY pgs=5590 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0ac000 tx=0x7fe75c0195a0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.399+0000 7fe7637fe640 1 -- 172.21.15.138:0/208754747 reap_dead start 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.403+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 shutdown_connections 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.403+0000 7fe769468640 1 --2- 172.21.15.138:0/208754747 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe76409eb10 0x7fe764136e10 unknown :-1 s=CLOSED pgs=5590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.403+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 >> 172.21.15.138:0/208754747 conn(0x7fe7640ab830 msgr2=0x7fe7640a9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.403+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 shutdown_connections 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.403+0000 7fe769468640 1 -- 172.21.15.138:0/208754747 wait complete. 2024-09-11T05:09:49.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.403+0000 7fe769468640 1 librados: shutdown 2024-09-11T05:09:49.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:09:49.455 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:09:49.456 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient: get_monmap_and_config 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient: build_initial_monmap 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient: monmap: 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:49.571426+0000 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:49.571426+0000 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.570 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:49.571 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:49.571 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:49.571 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:49.571 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:49.571 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 1 Processor -- start 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 1 -- start start 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient: init 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ea00a4a30 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6ea00a5520 0x7f6ea00a0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea00a5520 addr v2:172.21.15.190:3300/0 2024-09-11T05:09:49.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ea013dc00 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea006ef10 con 0x7f6ea00a4a30 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea006ecd0 con 0x7f6ea013dc00 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6ea00a3470 con 0x7f6ea00a5520 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.567+0000 7f6ea7ef9640 10 monclient(hunting): authenticate will time out at 1073.007886s 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea6496640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f6ea013dc00 0x7f6ea013fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.138:55974/0 (socket says 172.21.15.138:55974) 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5c95640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f6ea00a4a30 0x7f6ea00a4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:48862/0 (socket says 172.21.15.138:48862) 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea6496640 1 -- 172.21.15.138:0/1454196212 learned_addr learned my addr 172.21.15.138:0/1454196212 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 1 --2- 172.21.15.138:0/1454196212 >> v2:172.21.15.190:3300/0 conn(0x7f6ea00a5520 0x7f6ea00a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient(hunting): get_auth_request con 0x7f6ea00a5520 auth_method 0 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient(hunting): handle_auth_done global_id 20164 payload 274 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 1 -- 172.21.15.138:0/1454196212 >> v1:172.21.15.190:6789/0 conn(0x7f6ea013dc00 legacy=0x7f6ea013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 1 -- 172.21.15.138:0/1454196212 >> v1:172.21.15.138:6789/0 conn(0x7f6ea00a4a30 legacy=0x7f6ea00a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient: _finish_hunting 0 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 1 monclient: found mon.noname-c 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:09:49.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.571+0000 7f6ea5494640 1 -- 172.21.15.138:0/1454196212 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea006f110 con 0x7f6ea00a5520 2024-09-11T05:09:49.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea5494640 1 -- 172.21.15.138:0/1454196212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a5520 msgr2=0x7f6ea00a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea5494640 1 --2- 172.21.15.138:0/1454196212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a5520 0x7f6ea00a0850 secure :-1 s=READY pgs=5567 cs=0 l=1 rev1=1 crypto rx=0x7f6e900097c0 tx=0x7f6e9001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b52bc7c4c5e454f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 1 -- 172.21.15.138:0/1454196212 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e900b0070 con 0x7f6ea00a5520 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: dump: 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.579 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: _finish_auth 0 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: _check_auth_tickets 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 1 -- 172.21.15.138:0/1454196212 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e900ab600 con 0x7f6ea00a5520 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea7ef9640 5 monclient: authenticate success, global_id 20164 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 1 -- 172.21.15.138:0/1454196212 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e900ab9d0 con 0x7f6ea00a5520 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea4c93640 10 monclient: dump: 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea7ef9640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:49.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.575+0000 7f6ea7ef9640 4 set_mon_vals no callback set 2024-09-11T05:09:49.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.579+0000 7f6ea7ef9640 10 monclient: shutdown 2024-09-11T05:09:49.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.579+0000 7f6ea7ef9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:49.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.579+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/1454196212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a5520 msgr2=0x7f6ea00a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.579+0000 7f6ea7ef9640 1 --2- 172.21.15.138:0/1454196212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a5520 0x7f6ea00a0850 secure :-1 s=READY pgs=5567 cs=0 l=1 rev1=1 crypto rx=0x7f6e900097c0 tx=0x7f6e9001c330 comp rx=0 tx=0).stop 2024-09-11T05:09:49.586 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.583+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/1454196212 shutdown_connections 2024-09-11T05:09:49.586 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.583+0000 7f6ea7ef9640 1 --2- 172.21.15.138:0/1454196212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a5520 0x7f6ea00a0850 unknown :-1 s=CLOSED pgs=5567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.587 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.583+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/1454196212 >> 172.21.15.138:0/1454196212 conn(0x7f6ea00a6180 msgr2=0x7f6ea0063640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:49.587 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.583+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/1454196212 shutdown_connections 2024-09-11T05:09:49.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/1454196212 wait complete. 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 10 monclient: build_initial_monmap 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 10 monclient: monmap: 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:49.589 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:49.590 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 1 librados: starting msgr at 2024-09-11T05:09:49.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 1 librados: starting objecter 2024-09-11T05:09:49.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 1 Processor -- start 2024-09-11T05:09:49.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 1 -- start start 2024-09-11T05:09:49.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 1 librados: setting wanted keys 2024-09-11T05:09:49.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 1 librados: calling monclient init 2024-09-11T05:09:49.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.587+0000 7f6ea7ef9640 10 monclient: init 2024-09-11T05:09:49.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:49.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:09:49.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a4a30 0x7f6ea013c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea00a4a30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:49.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ea00a5520 0x7f6ea013c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea00a5520 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:49.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ea013dc00 0x7f6ea013cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea013dc00 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:49.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea0144140 con 0x7f6ea00a5520 2024-09-11T05:09:49.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea006ff50 con 0x7f6ea00a4a30 2024-09-11T05:09:49.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea01442c0 con 0x7f6ea013dc00 2024-09-11T05:09:49.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ea013dc00 0x7f6ea013cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ea013dc00 0x7f6ea013cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59094/0 (socket says 172.21.15.138:59094) 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 1 -- 172.21.15.138:0/4276495291 learned_addr learned my addr 172.21.15.138:0/4276495291 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea7ef9640 10 monclient(hunting): authenticate will time out at 1073.029707s 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 10 monclient(hunting): get_auth_request con 0x7f6ea013dc00 auth_method 0 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea5c95640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a4a30 0x7f6ea013c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea5c95640 10 monclient(hunting): get_auth_request con 0x7f6ea00a4a30 auth_method 0 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea5c95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea5c95640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea5c95640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 10 monclient(hunting): handle_auth_done global_id 22070 payload 824 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a4a30 msgr2=0x7f6ea013c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea00a4a30 0x7f6ea013c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.591+0000 7f6ea6496640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ea00a5520 msgr2=0x7f6ea013c9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6ea6496640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ea00a5520 0x7f6ea013c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6ea6496640 10 monclient: _finish_hunting 0 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6ea6496640 1 monclient: found mon.noname-c 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6ea6496640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:49.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6ea6496640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6ea6496640 1 -- 172.21.15.138:0/4276495291 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea00a0210 con 0x7f6ea013dc00 2024-09-11T05:09:49.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6ea6496640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ea013dc00 msgr2=0x7f6ea013cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6ea6496640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ea013dc00 0x7f6ea013cf20 secure :-1 s=READY pgs=5594 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c3c0 tx=0x7f6e9800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.595+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 client.?.objecter ms_handle_connect 0x7f6ea013dc00 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98017070 con 0x7f6ea013dc00 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: dump: 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.601 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: _finish_auth 0 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: _check_auth_tickets 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e980070d0 con 0x7f6ea013dc00 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e980074c0 con 0x7f6ea013dc00 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 monclient: dump: 2024-09-11T05:09:49.602 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 5 monclient: authenticate success, global_id 22070 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 10 monclient: _renew_subs 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea01262d0 con 0x7f6ea013dc00 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e97fff640 4 set_mon_vals no callback set 2024-09-11T05:09:49.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 10 client.22070.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:49.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 10 monclient: _renew_subs 2024-09-11T05:09:49.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea009ed50 con 0x7f6ea013dc00 2024-09-11T05:09:49.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 1 librados: init done 2024-09-11T05:09:49.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071480 0x7f6ea0071860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea009f390 con 0x7f6ea0071480 2024-09-11T05:09:49.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f6e9801b3b0 con 0x7f6ea013dc00 2024-09-11T05:09:49.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 1 --2- 172.21.15.138:0/4276495291 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6e8403cef0 0x7f6e8403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6e9804fb00 con 0x7f6ea013dc00 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 10 client.22070.objecter ms_dispatch 0x7f6ea0000ba0 osd_map(85..85 src has 1..85) 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 3 client.22070.objecter handle_osd_map got epochs [85,85] > 0 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 3 client.22070.objecter handle_osd_map decoding full epoch 85 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6e967fc640 20 client.22070.objecter dump_active .. 0 homeless 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.599+0000 7f6ea5494640 1 --2- 172.21.15.138:0/4276495291 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6e8403cef0 0x7f6e8403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5494640 10 monclient: get_auth_request con 0x7f6e8403cef0 auth_method 0 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071480 0x7f6ea0071860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 10 monclient: get_auth_request con 0x7f6ea0071480 auth_method 0 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6e967fc640 10 client.22070.objecter ms_handle_connect 0x7f6e8403cef0 2024-09-11T05:09:49.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5494640 1 --2- 172.21.15.138:0/4276495291 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6e8403cef0 0x7f6e8403f3b0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f6e900ab000 tx=0x7f6e90004740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 10 monclient(hunting): handle_auth_done global_id 20176 payload 824 2024-09-11T05:09:49.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.603+0000 7f6ea5c95640 1 -- 172.21.15.138:0/4276495291 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ea00a0850 con 0x7f6ea0071480 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6ea5c95640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071480 msgr2=0x7f6ea0071860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6ea5c95640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071480 0x7f6ea0071860 secure :-1 s=READY pgs=5571 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00e840 tx=0x7f6e9c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 10 client.22070.objecter ms_handle_connect 0x7f6ea0071480 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 10 client.22070.objecter resend_mon_ops 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c00caf0 con 0x7f6ea0071480 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 10 monclient: dump: 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.610 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.611 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.611 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.611 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.611 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.611 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.611 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.611 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6e9c018070 con 0x7f6ea0071480 2024-09-11T05:09:49.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:49.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071480 msgr2=0x7f6ea0071860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071480 0x7f6ea0071860 secure :-1 s=READY pgs=5571 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00e840 tx=0x7f6e9c00ed00 comp rx=0 tx=0).stop 2024-09-11T05:09:49.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071480 msgr2=0x7f6ea0071860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:49.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.607+0000 7f6e967fc640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071480 0x7f6ea0071860 secure :-1 s=CLOSED pgs=5571 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00e840 tx=0x7f6e9c00ed00 comp rx=0 tx=0).stop 2024-09-11T05:09:49.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.615+0000 7f6ea7ef9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:09:49.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.615+0000 7f6ea7ef9640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071da0 0x7f6ea0123a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.615+0000 7f6ea7ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.615+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea0141010 con 0x7f6ea0071da0 2024-09-11T05:09:49.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071da0 0x7f6ea0123a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 10 monclient: get_auth_request con 0x7f6ea0071da0 auth_method 0 2024-09-11T05:09:49.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 10 monclient(hunting): handle_auth_done global_id 20185 payload 824 2024-09-11T05:09:49.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 1 -- 172.21.15.138:0/4276495291 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ea0074b70 con 0x7f6ea0071da0 2024-09-11T05:09:49.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071da0 msgr2=0x7f6ea0123a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 10 client.22070.objecter ms_handle_connect 0x7f6ea0071da0 2024-09-11T05:09:49.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 10 client.22070.objecter resend_mon_ops 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea5c95640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071da0 0x7f6ea0123a80 secure :-1 s=READY pgs=5575 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00e840 tx=0x7f6e9c004cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c01de60 con 0x7f6ea0071da0 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 10 monclient: dump: 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.624 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f6e9c013420 con 0x7f6ea0071da0 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071da0 msgr2=0x7f6ea0123a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071da0 0x7f6ea0123a80 secure :-1 s=READY pgs=5575 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00e840 tx=0x7f6e9c004cc0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071da0 msgr2=0x7f6ea0123a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6e967fc640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea0071da0 0x7f6ea0123a80 unknown :-1 s=CLOSED pgs=5575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:09:49.625 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 14, 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 148353, 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 5, 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:09:49.626 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:09:49.627 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:09:49.628 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:09:49.629 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:09:49.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.631 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:09:49.632 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.633 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:09:49.634 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:09:49.635 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.636 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 9 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:09:49.637 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:09:49.638 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:09:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 10 librados: watch_flush enter 2024-09-11T05:09:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 10 librados: watch_flush exit 2024-09-11T05:09:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 20 client.22070.objecter shutdown clearing up homeless session... 2024-09-11T05:09:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 10 client.22070.objecter successfully canceled tick 2024-09-11T05:09:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6e8403cef0 msgr2=0x7f6e8403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 1 --2- 172.21.15.138:0/4276495291 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6e8403cef0 0x7f6e8403f3b0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f6e900ab000 tx=0x7f6e90004740 comp rx=0 tx=0).stop 2024-09-11T05:09:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 10 monclient: shutdown 2024-09-11T05:09:49.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ea013dc00 msgr2=0x7f6ea013cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 1 --2- 172.21.15.138:0/4276495291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ea013dc00 0x7f6ea013cf20 secure :-1 s=READY pgs=5594 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c3c0 tx=0x7f6e9800c880 comp rx=0 tx=0).stop 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea6496640 1 -- 172.21.15.138:0/4276495291 reap_dead start 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea6496640 1 -- 172.21.15.138:0/4276495291 reap_dead start 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 shutdown_connections 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.619+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 >> 172.21.15.138:0/4276495291 conn(0x7f6ea00a6180 msgr2=0x7f6ea00a86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.623+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 shutdown_connections 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.623+0000 7f6ea7ef9640 1 -- 172.21.15.138:0/4276495291 wait complete. 2024-09-11T05:09:49.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.623+0000 7f6ea7ef9640 1 librados: shutdown 2024-09-11T05:09:49.641 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:09:49.642 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T05:09:49.642 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T05:09:49.642 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T05:09:49.642 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T05:09:49.642 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T05:09:49.642 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:09:49.653 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T05:09:49.653 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:09:49.653 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:09:49.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient: get_monmap_and_config 2024-09-11T05:09:49.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient: build_initial_monmap 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient: monmap: 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:49.805615+0000 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:49.805615+0000 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:49.805 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:49.806 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:49.806 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:49.806 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:49.806 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 1 Processor -- start 2024-09-11T05:09:49.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 1 -- start start 2024-09-11T05:09:49.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient: init 2024-09-11T05:09:49.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f7c0a4030 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c0a4b20 0x7f4f7c0a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c0a4b20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f7c13da40 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c0a2850 con 0x7f4f7c0a4030 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c0a2b30 con 0x7f4f7c13da40 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c0a2e10 con 0x7f4f7c0a4b20 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f810b1640 10 monclient(hunting): authenticate will time out at 1073.241755s 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7ad9d640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7f4f7c0a4030 legacy=0x7f4f7c0a4410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:09:49.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c0a4b20 0x7f4f7c0a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c0a4b20 0x7f4f7c0a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59106/0 (socket says 172.21.15.138:59106) 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 1 -- 172.21.15.138:0/1480737066 learned_addr learned my addr 172.21.15.138:0/1480737066 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient(hunting): get_auth_request con 0x7f4f7c0a4b20 auth_method 0 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient(hunting): handle_auth_done global_id 22109 payload 274 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 1 -- 172.21.15.138:0/1480737066 >> v1:172.21.15.190:6789/0 conn(0x7f4f7c13da40 legacy=0x7f4f7c13fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 1 -- 172.21.15.138:0/1480737066 >> v1:172.21.15.138:6789/0 conn(0x7f4f7c0a4030 legacy=0x7f4f7c0a4410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient: _finish_hunting 0 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 1 monclient: found mon.noname-b 2024-09-11T05:09:49.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:49.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.803+0000 7f4f7a59c640 1 -- 172.21.15.138:0/1480737066 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c0a6c60 con 0x7f4f7c0a4b20 2024-09-11T05:09:49.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f7a59c640 1 -- 172.21.15.138:0/1480737066 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c0a4b20 msgr2=0x7f4f7c0a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f7a59c640 1 --2- 172.21.15.138:0/1480737066 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c0a4b20 0x7f4f7c0a3ab0 secure :-1 s=READY pgs=5598 cs=0 l=1 rev1=1 crypto rx=0x7f4f64009e80 tx=0x7f4f6401c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86b2f1ef018991cc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 1 -- 172.21.15.138:0/1480737066 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f640ac020 con 0x7f4f7c0a4b20 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: dump: 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.815 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: _finish_auth 0 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: _check_auth_tickets 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 5 monclient: authenticate success, global_id 22109 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 1 -- 172.21.15.138:0/1480737066 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f640a6440 con 0x7f4f7c0a4b20 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:49.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 4 set_mon_vals no callback set 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 1 -- 172.21.15.138:0/1480737066 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f640a67f0 con 0x7f4f7c0a4b20 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f79d9b640 10 monclient: dump: 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.817 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 10 monclient: shutdown 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 1 -- 172.21.15.138:0/1480737066 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c0a4b20 msgr2=0x7f4f7c0a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 1 --2- 172.21.15.138:0/1480737066 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c0a4b20 0x7f4f7c0a3ab0 secure :-1 s=READY pgs=5598 cs=0 l=1 rev1=1 crypto rx=0x7f4f64009e80 tx=0x7f4f6401c850 comp rx=0 tx=0).stop 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 1 -- 172.21.15.138:0/1480737066 shutdown_connections 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 1 --2- 172.21.15.138:0/1480737066 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c0a4b20 0x7f4f7c0a3ab0 unknown :-1 s=CLOSED pgs=5598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 1 -- 172.21.15.138:0/1480737066 >> 172.21.15.138:0/1480737066 conn(0x7f4f7c0ab940 msgr2=0x7f4f7c0a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 1 -- 172.21.15.138:0/1480737066 shutdown_connections 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.807+0000 7f4f810b1640 1 -- 172.21.15.138:0/1480737066 wait complete. 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient: build_initial_monmap 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient: monmap: 2024-09-11T05:09:49.818 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 librados: starting msgr at 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 librados: starting objecter 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 Processor -- start 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 -- start start 2024-09-11T05:09:49.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 librados: setting wanted keys 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 librados: calling monclient init 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient: init 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4f7c0a4030 0x7f4f7c124a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient(hunting): picked mon.noname-a con 0x7f4f7c0a4030 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0a4b20 0x7f4f7c13c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c0a4b20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c13da40 0x7f4f7c122450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7c13da40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c1441e0 con 0x7f4f7c0a4030 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c144060 con 0x7f4f7c0a4b20 2024-09-11T05:09:49.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:49.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c144360 con 0x7f4f7c13da40 2024-09-11T05:09:49.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:49.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f810b1640 10 monclient(hunting): authenticate will time out at 1073.249151s 2024-09-11T05:09:49.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7ad9d640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4f7c0a4030 msgr2=0x7f4f7c124a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:49.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7ad9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4f7c0a4030 0x7f4f7c124a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:49.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c13da40 0x7f4f7c122450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c13da40 0x7f4f7c122450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59108/0 (socket says 172.21.15.138:59108) 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 -- 172.21.15.138:0/3332510067 learned_addr learned my addr 172.21.15.138:0/3332510067 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient(hunting): get_auth_request con 0x7f4f7c13da40 auth_method 0 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:49.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient(hunting): handle_auth_done global_id 22112 payload 824 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7a59c640 1 --2- 172.21.15.138:0/3332510067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0a4b20 0x7f4f7c13c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 -- 172.21.15.138:0/3332510067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0a4b20 msgr2=0x7f4f7c13c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 --2- 172.21.15.138:0/3332510067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0a4b20 0x7f4f7c13c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 -- 172.21.15.138:0/3332510067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4f7c0a4030 msgr2=0x7f4f7c124a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 --2- 172.21.15.138:0/3332510067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4f7c0a4030 0x7f4f7c124a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient: _finish_hunting 0 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 monclient: found mon.noname-c 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 -- 172.21.15.138:0/3332510067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c1d1270 con 0x7f4f7c13da40 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 -- 172.21.15.138:0/3332510067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c13da40 msgr2=0x7f4f7c122450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f7b59e640 1 --2- 172.21.15.138:0/3332510067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c13da40 0x7f4f7c122450 secure :-1 s=READY pgs=5599 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00c360 tx=0x7f4f6c00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f737fe640 10 client.?.objecter ms_handle_connect 0x7f4f7c13da40 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.811+0000 7f4f737fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 1 -- 172.21.15.138:0/3332510067 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c017070 con 0x7f4f7c13da40 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: dump: 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.826 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: _finish_auth 0 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: _check_auth_tickets 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 5 monclient: authenticate success, global_id 22112 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 10 monclient: _renew_subs 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 1 -- 172.21.15.138:0/3332510067 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f6c0069e0 con 0x7f4f7c13da40 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7c140fc0 con 0x7f4f7c13da40 2024-09-11T05:09:49.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 1 -- 172.21.15.138:0/3332510067 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c006dd0 con 0x7f4f7c13da40 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 monclient: dump: 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:49.828 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 10 client.22112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f7959a640 4 set_mon_vals no callback set 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 10 monclient: _renew_subs 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f7c123f10 con 0x7f4f7c13da40 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 1 librados: init done 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f7c0a3ab0 con 0x7f4f7c13da40 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 1 -- 172.21.15.138:0/3332510067 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f4f6c007110 con 0x7f4f7c13da40 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 1 --2- 172.21.15.138:0/3332510067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f6003cea0 0x7f4f6003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:49.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 1 -- 172.21.15.138:0/3332510067 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4f6c04fb20 con 0x7f4f7c13da40 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 10 client.22112.objecter ms_dispatch 0x7f4f7c000ba0 osd_map(85..85 src has 1..85) 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 3 client.22112.objecter handle_osd_map got epochs [85,85] > 0 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 3 client.22112.objecter handle_osd_map decoding full epoch 85 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f737fe640 20 client.22112.objecter dump_active .. 0 homeless 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f7ad9d640 1 --2- 172.21.15.138:0/3332510067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f6003cea0 0x7f4f6003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.815+0000 7f4f7ad9d640 10 monclient: get_auth_request con 0x7f4f6003cea0 auth_method 0 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.819+0000 7f4f737fe640 10 client.22112.objecter ms_handle_connect 0x7f4f6003cea0 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.819+0000 7f4f7ad9d640 1 --2- 172.21.15.138:0/3332510067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f6003cea0 0x7f4f6003f360 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006fd0 tx=0x7f4f74006d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.819+0000 7f4f737fe640 1 -- 172.21.15.138:0/3332510067 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f4f6c0104a0 con 0x7f4f7c13da40 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.819+0000 7f4f737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:49.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:49.819+0000 7f4f737fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.055+0000 7f4f810b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:09:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.055+0000 7f4f810b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.055+0000 7f4f810b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.055+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f7c1c93f0 con 0x7f4f7c13da40 2024-09-11T05:09:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.055+0000 7f4f737fe640 1 -- 172.21.15.138:0/3332510067 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f4f6c019dd0 con 0x7f4f7c13da40 2024-09-11T05:09:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.055+0000 7f4f737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.055+0000 7f4f737fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:50.058 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:09:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 10 librados: watch_flush enter 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 10 librados: watch_flush exit 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 20 client.22112.objecter shutdown clearing up homeless session... 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 10 client.22112.objecter successfully canceled tick 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f6003cea0 msgr2=0x7f4f6003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 --2- 172.21.15.138:0/3332510067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f6003cea0 0x7f4f6003f360 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006fd0 tx=0x7f4f74006d40 comp rx=0 tx=0).stop 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 10 monclient: shutdown 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c13da40 msgr2=0x7f4f7c122450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 --2- 172.21.15.138:0/3332510067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c13da40 0x7f4f7c122450 secure :-1 s=READY pgs=5599 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00c360 tx=0x7f4f6c00c820 comp rx=0 tx=0).stop 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 shutdown_connections 2024-09-11T05:09:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 --2- 172.21.15.138:0/3332510067 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f6003cea0 0x7f4f6003f360 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 --2- 172.21.15.138:0/3332510067 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4f7c13da40 0x7f4f7c122450 unknown :-1 s=CLOSED pgs=5599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 --2- 172.21.15.138:0/3332510067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f7c0a4b20 0x7f4f7c13c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 --2- 172.21.15.138:0/3332510067 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4f7c0a4030 0x7f4f7c124a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 >> 172.21.15.138:0/3332510067 conn(0x7f4f7c0ab940 msgr2=0x7f4f7c06f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 shutdown_connections 2024-09-11T05:09:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 -- 172.21.15.138:0/3332510067 wait complete. 2024-09-11T05:09:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:50.059+0000 7f4f810b1640 1 librados: shutdown 2024-09-11T05:09:53.078 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.203+0000 7fbf8b2bd640 10 monclient: get_monmap_and_config 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.203+0000 7fbf8b2bd640 10 monclient: build_initial_monmap 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.203+0000 7fbf8b2bd640 10 monclient: monmap: 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:53.207378+0000 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:53.207378+0000 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:53.206 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:53.207 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:53.207 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:53.207 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:53.207 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:53.207 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:53.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.203+0000 7fbf8b2bd640 1 Processor -- start 2024-09-11T05:09:53.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.203+0000 7fbf8b2bd640 1 -- start start 2024-09-11T05:09:53.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.203+0000 7fbf8b2bd640 10 monclient: init 2024-09-11T05:09:53.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbf840a5b40 0x7fbf840a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient(hunting): picked mon.noname-a con 0x7fbf840a5b40 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf840a7130 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf8409d840 0x7fbf840a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient(hunting): picked mon.noname-b con 0x7fbf8409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8409d490 con 0x7fbf840a7130 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fbf8409e110 con 0x7fbf840a5b40 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbf840a69a0 con 0x7fbf8409d840 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:53.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8b2bd640 10 monclient(hunting): authenticate will time out at 1076.644283s 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf89059640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7fbf840a5b40 msgr2=0x7fbf840a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf88858640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7fbf840a7130 legacy=0x7fbf84138900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf89059640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbf840a5b40 0x7fbf840a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf8409d840 0x7fbf840a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf8409d840 0x7fbf840a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59118/0 (socket says 172.21.15.138:59118) 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 1 -- 172.21.15.138:0/656283322 learned_addr learned my addr 172.21.15.138:0/656283322 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient(hunting): get_auth_request con 0x7fbf8409d840 auth_method 0 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:53.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient(hunting): handle_auth_done global_id 22115 payload 274 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 1 -- 172.21.15.138:0/656283322 >> v2:172.21.15.138:3300/0 conn(0x7fbf840a5b40 msgr2=0x7fbf840a5f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 1 --2- 172.21.15.138:0/656283322 >> v2:172.21.15.138:3300/0 conn(0x7fbf840a5b40 0x7fbf840a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 1 -- 172.21.15.138:0/656283322 >> v1:172.21.15.138:6789/0 conn(0x7fbf840a7130 legacy=0x7fbf84138900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient: _finish_hunting 0 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 1 monclient: found mon.noname-b 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.207+0000 7fbf8985a640 1 -- 172.21.15.138:0/656283322 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf84070580 con 0x7fbf8409d840 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 -- 172.21.15.138:0/656283322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf8409d840 msgr2=0x7fbf840a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:53.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 --2- 172.21.15.138:0/656283322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf8409d840 0x7fbf840a6460 secure :-1 s=READY pgs=5600 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009870 tx=0x7fbf7c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc2ea0e441d3445d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:53.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 1 -- 172.21.15.138:0/656283322 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0b0070 con 0x7fbf8409d840 2024-09-11T05:09:53.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:53.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:09:53.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: dump: 2024-09-11T05:09:53.214 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:53.214 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:53.214 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:53.214 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: _finish_auth 0 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 1 -- 172.21.15.138:0/656283322 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf7c0ab600 con 0x7fbf8409d840 2024-09-11T05:09:53.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 1 -- 172.21.15.138:0/656283322 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0ab9d0 con 0x7fbf8409d840 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 5 monclient: authenticate success, global_id 22115 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 4 set_mon_vals no callback set 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf7bfff640 10 monclient: dump: 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:53.216 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient: shutdown 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/656283322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf8409d840 msgr2=0x7fbf840a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/656283322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf8409d840 0x7fbf840a6460 secure :-1 s=READY pgs=5600 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009870 tx=0x7fbf7c01c440 comp rx=0 tx=0).stop 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/656283322 shutdown_connections 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/656283322 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf8409d840 0x7fbf840a6460 unknown :-1 s=CLOSED pgs=5600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/656283322 >> v2:172.21.15.138:3300/0 conn(0x7fbf840a5b40 0x7fbf840a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/656283322 >> 172.21.15.138:0/656283322 conn(0x7fbf840aba10 msgr2=0x7fbf840a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:53.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/656283322 shutdown_connections 2024-09-11T05:09:53.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/656283322 wait complete. 2024-09-11T05:09:53.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient: build_initial_monmap 2024-09-11T05:09:53.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient: monmap: 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 librados: starting msgr at 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 librados: starting objecter 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 Processor -- start 2024-09-11T05:09:53.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- start start 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 librados: setting wanted keys 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 librados: calling monclient init 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient: init 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf8409d840 0x7fbf841d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient(hunting): picked mon.noname-b con 0x7fbf8409d840 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf840a5b40 0x7fbf84137a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient(hunting): picked mon.noname-a con 0x7fbf840a5b40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf840a7130 0x7fbf8413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient(hunting): picked mon.noname-c con 0x7fbf840a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:53.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbf84142f60 con 0x7fbf840a5b40 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbf84142de0 con 0x7fbf8409d840 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbf841430e0 con 0x7fbf840a7130 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8b2bd640 10 monclient(hunting): authenticate will time out at 1076.651577s 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf88858640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf840a5b40 msgr2=0x7fbf84137a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf88858640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf840a5b40 0x7fbf84137a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf840a7130 0x7fbf8413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf840a7130 0x7fbf8413d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59134/0 (socket says 172.21.15.138:59134) 2024-09-11T05:09:53.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 -- 172.21.15.138:0/843020965 learned_addr learned my addr 172.21.15.138:0/843020965 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient(hunting): get_auth_request con 0x7fbf840a7130 auth_method 0 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient(hunting): handle_auth_done global_id 22118 payload 824 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 -- 172.21.15.138:0/843020965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf8409d840 msgr2=0x7fbf841d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 --2- 172.21.15.138:0/843020965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf8409d840 0x7fbf841d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 -- 172.21.15.138:0/843020965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf840a5b40 msgr2=0x7fbf84137a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 --2- 172.21.15.138:0/843020965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf840a5b40 0x7fbf84137a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient: _finish_hunting 0 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 monclient: found mon.noname-c 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.211+0000 7fbf8985a640 1 -- 172.21.15.138:0/843020965 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8413ff90 con 0x7fbf840a7130 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8985a640 1 -- 172.21.15.138:0/843020965 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf840a7130 msgr2=0x7fbf8413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8985a640 1 --2- 172.21.15.138:0/843020965 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf840a7130 0x7fbf8413d940 secure :-1 s=READY pgs=5601 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c01cce0 tx=0x7fbf7c01ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 client.?.objecter ms_handle_connect 0x7fbf840a7130 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 1 -- 172.21.15.138:0/843020965 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0ba020 con 0x7fbf840a7130 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: dump: 2024-09-11T05:09:53.226 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: _finish_auth 0 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 1 -- 172.21.15.138:0/843020965 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf7c0b6630 con 0x7fbf840a7130 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 5 monclient: authenticate success, global_id 22118 2024-09-11T05:09:53.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 10 monclient: _renew_subs 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8413f430 con 0x7fbf840a7130 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 1 -- 172.21.15.138:0/843020965 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0b0040 con 0x7fbf840a7130 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 monclient: dump: 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:53.228 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf7affd640 4 set_mon_vals no callback set 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 10 client.22118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 10 monclient: _renew_subs 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf84121e50 con 0x7fbf840a7130 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 1 librados: init done 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf84138900 con 0x7fbf840a7130 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 1 -- 172.21.15.138:0/843020965 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fbf7c0b3030 con 0x7fbf840a7130 2024-09-11T05:09:53.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 1 --2- 172.21.15.138:0/843020965 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbf6803ce50 0x7fbf6803f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 1 -- 172.21.15.138:0/843020965 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbf7c0ea870 con 0x7fbf840a7130 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 10 client.22118.objecter ms_dispatch 0x7fbf84000ba0 osd_map(85..85 src has 1..85) 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 3 client.22118.objecter handle_osd_map got epochs [85,85] > 0 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 3 client.22118.objecter handle_osd_map decoding full epoch 85 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.215+0000 7fbf79ffb640 20 client.22118.objecter dump_active .. 0 homeless 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.219+0000 7fbf89059640 1 --2- 172.21.15.138:0/843020965 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbf6803ce50 0x7fbf6803f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.219+0000 7fbf89059640 10 monclient: get_auth_request con 0x7fbf6803ce50 auth_method 0 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.219+0000 7fbf79ffb640 10 client.22118.objecter ms_handle_connect 0x7fbf6803ce50 2024-09-11T05:09:53.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.219+0000 7fbf89059640 1 --2- 172.21.15.138:0/843020965 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbf6803ce50 0x7fbf6803f310 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7fbf80003d10 tx=0x7fbf80009230 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:53.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.223+0000 7fbf79ffb640 1 -- 172.21.15.138:0/843020965 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbf7c0c4030 con 0x7fbf840a7130 2024-09-11T05:09:53.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.223+0000 7fbf79ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:09:53.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.223+0000 7fbf79ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:09:53.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.459+0000 7fbf8b2bd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:09:53.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.459+0000 7fbf8b2bd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:53.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.459+0000 7fbf8b2bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:09:53.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.459+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbf84069fa0 con 0x7fbf840a7130 2024-09-11T05:09:53.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.459+0000 7fbf79ffb640 1 -- 172.21.15.138:0/843020965 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fbf7c016150 con 0x7fbf840a7130 2024-09-11T05:09:53.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.459+0000 7fbf79ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:09:53.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.459+0000 7fbf79ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:09:53.463 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:09:53.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 10 librados: watch_flush enter 2024-09-11T05:09:53.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 10 librados: watch_flush exit 2024-09-11T05:09:53.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 20 client.22118.objecter shutdown clearing up homeless session... 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 10 client.22118.objecter successfully canceled tick 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbf6803ce50 msgr2=0x7fbf6803f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/843020965 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbf6803ce50 0x7fbf6803f310 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 crypto rx=0x7fbf80003d10 tx=0x7fbf80009230 comp rx=0 tx=0).stop 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 10 monclient: shutdown 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf840a7130 msgr2=0x7fbf8413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/843020965 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf840a7130 0x7fbf8413d940 secure :-1 s=READY pgs=5601 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c01cce0 tx=0x7fbf7c01ac80 comp rx=0 tx=0).stop 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 shutdown_connections 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/843020965 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbf6803ce50 0x7fbf6803f310 unknown :-1 s=CLOSED pgs=3285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/843020965 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbf840a7130 0x7fbf8413d940 unknown :-1 s=CLOSED pgs=5601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/843020965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbf840a5b40 0x7fbf84137a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 --2- 172.21.15.138:0/843020965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbf8409d840 0x7fbf841d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 >> 172.21.15.138:0/843020965 conn(0x7fbf840aba10 msgr2=0x7fbf840a4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:53.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 shutdown_connections 2024-09-11T05:09:53.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 -- 172.21.15.138:0/843020965 wait complete. 2024-09-11T05:09:53.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:53.463+0000 7fbf8b2bd640 1 librados: shutdown 2024-09-11T05:09:56.483 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient: get_monmap_and_config 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient: build_initial_monmap 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient: monmap: 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:09:56.615114+0000 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:09:56.615114+0000 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:56.614 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:56.615 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:09:56.615 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:09:56.615 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:09:56.615 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:09:56.615 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:09:56.615 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:56.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 1 Processor -- start 2024-09-11T05:09:56.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 1 -- start start 2024-09-11T05:09:56.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient: init 2024-09-11T05:09:56.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:56.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-11T05:09:56.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 0x7f674c09e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:56.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient(hunting): picked mon.noname-a con 0x7f674c06e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:09:56.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f674c09e960 addr v1:172.21.15.138:6789/0 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f674c09ed70 addr v1:172.21.15.190:6789/0 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674c136510 con 0x7f674c09e960 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674c09f540 con 0x7f674c09ed70 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f674c06e4c0 con 0x7f674c06e0e0 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6752853640 10 monclient(hunting): authenticate will time out at 1080.051252s 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f674b7fe640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7f674c09e960 legacy=0x7f674c1356f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f674bfff640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 msgr2=0x7f674c09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f674bfff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 0x7f674c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6750df0640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f674c09ed70 0x7f674c135e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.138:55994/0 (socket says 172.21.15.138:55994) 2024-09-11T05:09:56.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.611+0000 7f6750df0640 1 -- 172.21.15.138:0/1466576083 learned_addr learned my addr 172.21.15.138:0/1466576083 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:09:56.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.615+0000 7f6750df0640 1 -- 172.21.15.138:0/1466576083 >> v1:172.21.15.190:6789/0 conn(0x7f674c09ed70 legacy=0x7f674c135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:56.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.815+0000 7f674bfff640 1 -- 172.21.15.138:0/1466576083 >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 msgr2=0x7f674c09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:56.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.815+0000 7f674b7fe640 1 -- 172.21.15.138:0/1466576083 >> v1:172.21.15.138:6789/0 conn(0x7f674c09e960 legacy=0x7f674c1356f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:09:56.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:56.815+0000 7f674bfff640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 0x7f674c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:09:57.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:57.219+0000 7f674bfff640 1 -- 172.21.15.138:0/1466576083 >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 msgr2=0x7f674c09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:57.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:57.219+0000 7f674b7fe640 1 -- 172.21.15.138:0/1466576083 >> v1:172.21.15.138:6789/0 conn(0x7f674c09e960 legacy=0x7f674c1356f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:09:57.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:57.219+0000 7f674bfff640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 0x7f674c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:09:58.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:58.023+0000 7f674bfff640 1 -- 172.21.15.138:0/1466576083 >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 msgr2=0x7f674c09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:58.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:58.023+0000 7f674b7fe640 1 -- 172.21.15.138:0/1466576083 >> v1:172.21.15.138:6789/0 conn(0x7f674c09e960 legacy=0x7f674c1356f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:09:58.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:58.023+0000 7f674bfff640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 0x7f674c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:09:59.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient(hunting): tick 2024-09-11T05:09:59.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:09:59.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 monclient(hunting): continuing hunt 2024-09-11T05:09:59.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:09:59.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 -- 172.21.15.138:0/1466576083 >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 msgr2=0x7f674c09e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.138:3300/0 conn(0x7f674c06e0e0 0x7f674c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 -- 172.21.15.138:0/1466576083 >> v1:172.21.15.190:6789/0 conn(0x7f674c09ed70 legacy=0x7f674c135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 -- 172.21.15.138:0/1466576083 >> v1:172.21.15.138:6789/0 conn(0x7f674c09e960 legacy=0x7f674c1356f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient: _add_conns ranks=[2,4] 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 --2- 172.21.15.138:0/1466576083 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f672c003780 0x7f672c005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f672c003780 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.190:3300/0 conn(0x7f672c006210 0x7f672c0086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f672c006210 addr v2:172.21.15.190:3300/0 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 -- 172.21.15.138:0/1466576083 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f674c06e4c0 con 0x7f672c006210 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 1 -- 172.21.15.138:0/1466576083 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f672c008d90 con 0x7f672c003780 2024-09-11T05:09:59.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f6749ffb640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 1 --2- 172.21.15.138:0/1466576083 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f672c003780 0x7f672c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient(hunting): get_auth_request con 0x7f672c003780 auth_method 0 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.190:3300/0 conn(0x7f672c006210 0x7f672c0086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 10 monclient(hunting): get_auth_request con 0x7f672c006210 auth_method 0 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:59.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:59.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 1 -- 172.21.15.138:0/1466576083 >> v2:172.21.15.190:3300/0 conn(0x7f672c006210 msgr2=0x7f672c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:09:59.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 1 -- 172.21.15.138:0/1466576083 >> v2:172.21.15.190:3300/0 conn(0x7f672c006210 msgr2=0x7f672c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:09:59.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.190:3300/0 conn(0x7f672c006210 0x7f672c0086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:09:59.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674b7fe640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.190:3300/0 conn(0x7f672c006210 0x7f672c0086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:59.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient(hunting): handle_auth_done global_id 22121 payload 274 2024-09-11T05:09:59.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 1 -- 172.21.15.138:0/1466576083 >> v2:172.21.15.190:3300/0 conn(0x7f672c006210 msgr2=0x7f672c0086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:09:59.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 1 --2- 172.21.15.138:0/1466576083 >> v2:172.21.15.190:3300/0 conn(0x7f672c006210 0x7f672c0086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:09:59.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:09:59.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 1 monclient: found mon.noname-b 2024-09-11T05:09:59.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:09:59.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:09:59.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.615+0000 7f674bfff640 1 -- 172.21.15.138:0/1466576083 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c09fb00 con 0x7f672c003780 2024-09-11T05:09:59.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.623+0000 7f674bfff640 1 -- 172.21.15.138:0/1466576083 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f672c003780 msgr2=0x7f672c005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:09:59.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.623+0000 7f674bfff640 1 --2- 172.21.15.138:0/1466576083 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f672c003780 0x7f672c005bd0 secure :-1 s=READY pgs=5602 cs=0 l=1 rev1=1 crypto rx=0x7f6734003990 tx=0x7f673401cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76e8e2808b3afcdc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:09:59.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 1 -- 172.21.15.138:0/1466576083 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6734006e60 con 0x7f672c003780 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: dump: 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:59.639 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: _finish_auth 0 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: _check_auth_tickets 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 1 -- 172.21.15.138:0/1466576083 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67340a9070 con 0x7f672c003780 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 1 -- 172.21.15.138:0/1466576083 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340a5ce0 con 0x7f672c003780 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f674affd640 10 monclient: dump: 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:59.640 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 5 monclient: authenticate success, global_id 22121 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 10 monclient: get_monmap_and_config success 2024-09-11T05:09:59.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 4 set_mon_vals no callback set 2024-09-11T05:09:59.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 10 monclient: shutdown 2024-09-11T05:09:59.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:09:59.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 1 -- 172.21.15.138:0/1466576083 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f672c003780 msgr2=0x7f672c005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:09:59.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 1 --2- 172.21.15.138:0/1466576083 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f672c003780 0x7f672c005bd0 secure :-1 s=READY pgs=5602 cs=0 l=1 rev1=1 crypto rx=0x7f6734003990 tx=0x7f673401cb40 comp rx=0 tx=0).stop 2024-09-11T05:09:59.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6750df0640 1 -- 172.21.15.138:0/1466576083 reap_dead start 2024-09-11T05:09:59.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 1 -- 172.21.15.138:0/1466576083 shutdown_connections 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.635+0000 7f6752853640 1 -- 172.21.15.138:0/1466576083 >> 172.21.15.138:0/1466576083 conn(0x7f674c063640 msgr2=0x7f674c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 -- 172.21.15.138:0/1466576083 shutdown_connections 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 -- 172.21.15.138:0/1466576083 wait complete. 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient: build_initial_monmap 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient: monmap: 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:09:59.646 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 librados: starting msgr at 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 librados: starting objecter 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 Processor -- start 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 -- start start 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 librados: setting wanted keys 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 librados: calling monclient init 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient: init 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient: _reopen_session rank -1 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f674c134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient(hunting): picked mon.noname-a con 0x7f672c003780 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:59.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient(hunting): picked mon.noname-b con 0x7f674c135250 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674bfff640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f674c134d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f674c134d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient(hunting): picked mon.noname-c con 0x7f674c11d570 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0a1430 con 0x7f672c003780 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0a12b0 con 0x7f674c135250 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient(hunting): start opening mon connection 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f674c141590 con 0x7f674c11d570 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient(hunting): _renew_subs 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6752853640 10 monclient(hunting): authenticate will time out at 1083.079118s 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:59.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:45730/0 (socket says 172.21.15.138:45730) 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 learned_addr learned my addr 172.21.15.138:0/2448711790 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:09:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:59.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:09:59.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:09:59.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:09:59.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.639+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:09:59.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f674c134d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:09:59.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f674c134d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:09:59.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:09:59.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:09:59.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:09:59.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:09:59.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:09:59.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:09:59.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:09:59.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:09:59.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:09:59.843+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:10:00.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f674c134d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:00.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f674c134d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:00.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:10:00.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:00.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:00.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:00.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:10:00.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:00.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:00.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:00.247+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:10:01.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f674c134d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f674c134d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6748ff9640 1 -- 172.21.15.138:0/2448711790 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:01.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:01.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:01.051+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:10:02.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): tick 2024-09-11T05:10:02.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:10:02.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 monclient(hunting): continuing hunt 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f674c11da00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f674c11da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f674c1356b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f674c1356b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f674c134d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f674c134d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): picked mon.noname-c con 0x7f674c11d570 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f6730007ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): picked mon.noname-a con 0x7f672c003780 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:10:02.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): picked mon.noname-b con 0x7f674c135250 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0a1430 con 0x7f672c003780 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0a12b0 con 0x7f674c135250 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f674c141590 con 0x7f674c11d570 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f67437fe640 10 monclient(hunting): _renew_subs 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f6730007ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:02.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f6730007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f6750df0640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f6750df0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f6750df0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f6750df0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.639+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:10:02.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.643+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:10:02.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.643+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:02.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.643+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:02.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.643+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:10:02.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:02.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:10:02.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:02.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:02.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:02.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:02.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:02.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:02.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:10:02.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f6730007ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:02.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.843+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f6730007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:10:02.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:02.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:10:02.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:02.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:02.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:02.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:02.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:02.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:02.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:02.847+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:10:03.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f6730007ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:03.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:03.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f6730007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:10:03.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:10:03.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:03.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:03.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:03.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:03.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:03.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:03.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.247+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:03.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:03.251+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:10:04.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674b7fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f6730007ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:04.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f6730007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:10:04.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:04.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:10:04.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:04.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:04.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:04.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:04.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:04.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:04.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.051+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:10:04.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:10:04.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:04.055+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:10:04.770 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: Running main() from gmock_main.cc 2024-09-11T05:10:04.770 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-11T05:10:04.770 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] Global test environment set-up. 2024-09-11T05:10:04.770 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4869 ms) 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5038 ms) 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3996 ms) 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (4452 ms) 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-11T05:10:04.771 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (6984 ms) 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25339 ms total) 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5038 ms) 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5976 ms) 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8030 ms) 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6014 ms) 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4027 ms) 2024-09-11T05:10:04.772 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-11T05:10:04.773 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi190-29023-11. 2024-09-11T05:10:04.773 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: Waiting for snaps to purge. 2024-09-11T05:10:04.773 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (24358 ms) 2024-09-11T05:10:04.773 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (53443 ms total) 2024-09-11T05:10:04.773 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: 2024-09-11T05:10:04.773 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] Global test environment tear-down 2024-09-11T05:10:04.773 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (78783 ms total) 2024-09-11T05:10:04.773 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-11T05:10:04.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-11T05:10:04.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] Global test environment set-up. 2024-09-11T05:10:04.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-11T05:10:04.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-11T05:10:04.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-11T05:10:04.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-11T05:10:04.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T05:10:04.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T05:08:45.714+0000 7f327dd7c880 -1 failed for service _ceph-mon._tcp 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T05:08:45.714+0000 7f327dd7c880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (62 ms) 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5011 ms) 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5074 ms total) 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: started 0x559eb5407df0 2024-09-11T05:10:04.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: started 0x559eb5627000 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: started 2 aios 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: waiting 0x559eb5407df0 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: waiting 0x559eb5627000 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: done. 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5185 ms) 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5185 ms total) 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.Exec (146 ms) 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (17 ms) 2024-09-11T05:10:04.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (7 ms) 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5814 ms) 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (1 ms) 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (46437 ms) 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 8 tests from LibRadosMisc (52422 ms total) 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] Global test environment tear-down 2024-09-11T05:10:04.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (79117 ms total) 2024-09-11T05:10:04.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-11T05:10:05.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): tick 2024-09-11T05:10:05.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:10:05.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 monclient(hunting): continuing hunt 2024-09-11T05:10:05.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:10:05.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f6730007ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f6730007ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): picked mon.noname-c con 0x7f674c11d570 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): picked mon.noname-b con 0x7f674c135250 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f673000aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): picked mon.noname-a con 0x7f672c003780 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:05.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0a1430 con 0x7f672c003780 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0a12b0 con 0x7f674c135250 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f674c141590 con 0x7f674c11d570 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f67437fe640 10 monclient(hunting): _renew_subs 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f6750df0640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f673000aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.639+0000 7f6750df0640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f673000aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient(hunting): get_auth_request con 0x7f674c11d570 auth_method 0 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:05.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674b7fe640 10 monclient(hunting): get_auth_request con 0x7f674c135250 auth_method 0 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient(hunting): handle_auth_done global_id 22124 payload 824 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 msgr2=0x7f673000a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 msgr2=0x7f673000aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:05.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f673000aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:10:05.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 monclient: found mon.noname-c 2024-09-11T05:10:05.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:10:05.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:10:05.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c11fe60 con 0x7f674c11d570 2024-09-11T05:10:05.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:05.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f674bfff640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 secure :-1 s=READY pgs=5604 cs=0 l=1 rev1=1 crypto rx=0x7f67340ad000 tx=0x7f67340198e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:05.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f6748ff9640 10 client.?.objecter ms_handle_connect 0x7f674c11d570 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f6748ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f6748ff9640 1 -- 172.21.15.138:0/2448711790 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340c1070 con 0x7f674c11d570 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f6748ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f6748ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.643+0000 7f6748ff9640 10 monclient: dump: 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:05.652 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 10 monclient: _finish_auth 0 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 1 -- 172.21.15.138:0/2448711790 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67340ae890 con 0x7f674c11d570 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 5 monclient: authenticate success, global_id 22124 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 10 monclient: _renew_subs 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c09e110 con 0x7f674c11d570 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:10:05.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 1 -- 172.21.15.138:0/2448711790 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340aec40 con 0x7f674c11d570 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 10 monclient: dump: 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6749ffb640 4 set_mon_vals no callback set 2024-09-11T05:10:05.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 10 client.22124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 10 monclient: _renew_subs 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c13dee0 con 0x7f674c11d570 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 1 librados: init done 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 1 -- 172.21.15.138:0/2448711790 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f67340bd430 con 0x7f674c11d570 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674c09e420 con 0x7f674c11d570 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 1 --2- 172.21.15.138:0/2448711790 >> v2:172.21.15.190:6800/1562782189 conn(0x7f672c03dbd0 0x7f672c040090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 1 -- 172.21.15.138:0/2448711790 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(86..86 src has 1..86) ==== 7173+0+0 (secure 0 0 0) 0x7f67340ec470 con 0x7f674c11d570 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 10 client.22124.objecter ms_dispatch 0x7f674c000ba0 osd_map(86..86 src has 1..86) 2024-09-11T05:10:05.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 3 client.22124.objecter handle_osd_map got epochs [86,86] > 0 2024-09-11T05:10:05.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 3 client.22124.objecter handle_osd_map decoding full epoch 86 2024-09-11T05:10:05.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f6748ff9640 20 client.22124.objecter dump_active .. 0 homeless 2024-09-11T05:10:05.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> v2:172.21.15.190:6800/1562782189 conn(0x7f672c03dbd0 0x7f672c040090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:05.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.647+0000 7f674b7fe640 10 monclient: get_auth_request con 0x7f672c03dbd0 auth_method 0 2024-09-11T05:10:05.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.651+0000 7f674b7fe640 1 --2- 172.21.15.138:0/2448711790 >> v2:172.21.15.190:6800/1562782189 conn(0x7f672c03dbd0 0x7f672c040090 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f6744000eb0 tx=0x7f674400d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:05.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.651+0000 7f6748ff9640 10 client.22124.objecter ms_handle_connect 0x7f672c03dbd0 2024-09-11T05:10:05.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.655+0000 7f6748ff9640 1 -- 172.21.15.138:0/2448711790 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f67340167c0 con 0x7f674c11d570 2024-09-11T05:10:05.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.655+0000 7f6748ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:10:05.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.655+0000 7f6748ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:10:05.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.923+0000 7f6752853640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:10:05.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.923+0000 7f6752853640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:10:05.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.923+0000 7f6752853640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:10:05.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.923+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f674c09dea0 con 0x7f674c11d570 2024-09-11T05:10:05.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.923+0000 7f6748ff9640 1 -- 172.21.15.138:0/2448711790 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f67340ea510 con 0x7f674c11d570 2024-09-11T05:10:05.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.923+0000 7f6748ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:10:05.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.923+0000 7f6748ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:10:05.928 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:10:05.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 10 librados: watch_flush enter 2024-09-11T05:10:05.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 10 librados: watch_flush exit 2024-09-11T05:10:05.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 20 client.22124.objecter shutdown clearing up homeless session... 2024-09-11T05:10:05.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 10 client.22124.objecter successfully canceled tick 2024-09-11T05:10:05.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 >> v2:172.21.15.190:6800/1562782189 conn(0x7f672c03dbd0 msgr2=0x7f672c040090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:05.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 --2- 172.21.15.138:0/2448711790 >> v2:172.21.15.190:6800/1562782189 conn(0x7f672c03dbd0 0x7f672c040090 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f6744000eb0 tx=0x7f674400d040 comp rx=0 tx=0).stop 2024-09-11T05:10:05.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 10 monclient: shutdown 2024-09-11T05:10:05.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:10:05.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 msgr2=0x7f67300056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:05.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 secure :-1 s=READY pgs=5604 cs=0 l=1 rev1=1 crypto rx=0x7f67340ad000 tx=0x7f67340198e0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 shutdown_connections 2024-09-11T05:10:05.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 --2- 172.21.15.138:0/2448711790 >> v2:172.21.15.190:6800/1562782189 conn(0x7f672c03dbd0 0x7f672c040090 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f672c003780 0x7f673000aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f674c135250 0x7f673000a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 --2- 172.21.15.138:0/2448711790 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f674c11d570 0x7f67300056f0 unknown :-1 s=CLOSED pgs=5604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:05.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 >> 172.21.15.138:0/2448711790 conn(0x7f674c063640 msgr2=0x7f674c073ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:10:05.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 shutdown_connections 2024-09-11T05:10:05.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 -- 172.21.15.138:0/2448711790 wait complete. 2024-09-11T05:10:05.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:05.927+0000 7f6752853640 1 librados: shutdown 2024-09-11T05:10:05.947 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:10:05.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T05:10:05.947 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:10:06.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient: get_monmap_and_config 2024-09-11T05:10:06.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient: build_initial_monmap 2024-09-11T05:10:06.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient: monmap: 2024-09-11T05:10:06.068 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:10:06.068 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:10:06.069598+0000 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:10:06.069598+0000 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:10:06.069 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 1 Processor -- start 2024-09-11T05:10:06.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 1 -- start start 2024-09-11T05:10:06.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient: init 2024-09-11T05:10:06.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient: _reopen_session rank -1 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8774070010 addr v1:172.21.15.138:6789/0 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f877409e6e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409eb10 0x7f8774140d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient(hunting): picked mon.noname-b con 0x7f877409eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8774135b60 con 0x7f8774070010 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87740a8990 con 0x7f877409e6e0 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f87740703f0 con 0x7f877409eb10 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient(hunting): _renew_subs 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877a2f1640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7f8774070010 legacy=0x7f8774135450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877c555640 10 monclient(hunting): authenticate will time out at 1089.505739s 2024-09-11T05:10:06.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409eb10 0x7f8774140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409eb10 0x7f8774140d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:45832/0 (socket says 172.21.15.138:45832) 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 1 -- 172.21.15.138:0/723400905 learned_addr learned my addr 172.21.15.138:0/723400905 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient(hunting): get_auth_request con 0x7f877409eb10 auth_method 0 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient(hunting): handle_auth_done global_id 22127 payload 274 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 1 -- 172.21.15.138:0/723400905 >> v1:172.21.15.190:6789/0 conn(0x7f877409e6e0 legacy=0x7f877413d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 1 -- 172.21.15.138:0/723400905 >> v1:172.21.15.138:6789/0 conn(0x7f8774070010 legacy=0x7f8774135450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:10:06.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient: _finish_hunting 0 2024-09-11T05:10:06.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 1 monclient: found mon.noname-b 2024-09-11T05:10:06.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:10:06.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:10:06.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.067+0000 7f877aaf2640 1 -- 172.21.15.138:0/723400905 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877409cbf0 con 0x7f877409eb10 2024-09-11T05:10:06.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f877aaf2640 1 -- 172.21.15.138:0/723400905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409eb10 msgr2=0x7f8774140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:06.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f877aaf2640 1 --2- 172.21.15.138:0/723400905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409eb10 0x7f8774140d70 secure :-1 s=READY pgs=5605 cs=0 l=1 rev1=1 crypto rx=0x7f876c009e80 tx=0x7f876c01c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8337f94392bdea27 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 1 -- 172.21.15.138:0/723400905 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0a56c0 con 0x7f877409eb10 2024-09-11T05:10:06.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: dump: 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.077 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: _finish_auth 0 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: _check_auth_tickets 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 1 -- 172.21.15.138:0/723400905 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f876c0a5860 con 0x7f877409eb10 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f877c555640 5 monclient: authenticate success, global_id 22127 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f877c555640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: handle_config config(0 keys) 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 1 -- 172.21.15.138:0/723400905 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0a5c30 con 0x7f877409eb10 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f87792ef640 10 monclient: dump: 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.078 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f877c555640 10 monclient: get_monmap_and_config success 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.071+0000 7f877c555640 4 set_mon_vals no callback set 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient: shutdown 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- 172.21.15.138:0/723400905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409eb10 msgr2=0x7f8774140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 --2- 172.21.15.138:0/723400905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409eb10 0x7f8774140d70 secure :-1 s=READY pgs=5605 cs=0 l=1 rev1=1 crypto rx=0x7f876c009e80 tx=0x7f876c01c8e0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- 172.21.15.138:0/723400905 shutdown_connections 2024-09-11T05:10:06.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 --2- 172.21.15.138:0/723400905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409eb10 0x7f8774140d70 unknown :-1 s=CLOSED pgs=5605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- 172.21.15.138:0/723400905 >> 172.21.15.138:0/723400905 conn(0x7f87740ab830 msgr2=0x7f87740a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:10:06.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- 172.21.15.138:0/723400905 shutdown_connections 2024-09-11T05:10:06.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- 172.21.15.138:0/723400905 wait complete. 2024-09-11T05:10:06.086 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient: build_initial_monmap 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient: monmap: 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 librados: starting msgr at 2024-09-11T05:10:06.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 librados: starting objecter 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 Processor -- start 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- start start 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 librados: setting wanted keys 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 librados: calling monclient init 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient: init 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient: _reopen_session rank -1 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774070010 0x7f87741d0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient(hunting): picked mon.noname-b con 0x7f8774070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409e6e0 0x7f8774140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient(hunting): picked mon.noname-c con 0x7f877409e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f877409eb10 0x7f8774136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient(hunting): picked mon.noname-a con 0x7f877409eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8774143ec0 con 0x7f877409eb10 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8774143d40 con 0x7f8774070010 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8774144040 con 0x7f877409e6e0 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient(hunting): _renew_subs 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877c555640 10 monclient(hunting): authenticate will time out at 1089.513779s 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877aaf2640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f877409eb10 msgr2=0x7f8774136e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877aaf2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f877409eb10 0x7f8774136e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409e6e0 0x7f8774140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409e6e0 0x7f8774140430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:45842/0 (socket says 172.21.15.138:45842) 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 -- 172.21.15.138:0/2883319791 learned_addr learned my addr 172.21.15.138:0/2883319791 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient(hunting): get_auth_request con 0x7f877409e6e0 auth_method 0 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877a2f1640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774070010 0x7f87741d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient(hunting): handle_auth_done global_id 22130 payload 824 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774070010 msgr2=0x7f87741d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774070010 0x7f87741d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f877409eb10 msgr2=0x7f8774136e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f877409eb10 0x7f8774136e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient: _finish_hunting 0 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 monclient: found mon.noname-c 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f8779af0640 1 -- 172.21.15.138:0/2883319791 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8774138930 con 0x7f877409e6e0 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877a2f1640 10 monclient: get_auth_request con 0x7f8774070010 auth_method 0 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.075+0000 7f877a2f1640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8774070010 0x7f87741d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f8779af0640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409e6e0 msgr2=0x7f8774140430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f8779af0640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409e6e0 0x7f8774140430 secure :-1 s=READY pgs=5606 cs=0 l=1 rev1=1 crypto rx=0x7f876400c450 tx=0x7f876400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 client.?.objecter ms_handle_connect 0x7f877409e6e0 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8764012a40 con 0x7f877409e6e0 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: dump: 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: _finish_auth 0 2024-09-11T05:10:06.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: _check_auth_tickets 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 5 monclient: authenticate success, global_id 22130 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8764017070 con 0x7f877409e6e0 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 10 monclient: _renew_subs 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f877413ac90 con 0x7f877409e6e0 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8764012e40 con 0x7f877409e6e0 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 monclient: dump: 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.093 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f8778aee640 4 set_mon_vals no callback set 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 10 client.22130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 10 monclient: _renew_subs 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87740a2330 con 0x7f877409e6e0 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 1 librados: init done 2024-09-11T05:10:06.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8774140970 0x7f87741373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f877413a880 con 0x7f8774140970 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8774140970 0x7f87741373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 10 monclient: get_auth_request con 0x7f8774140970 auth_method 0 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 10 monclient(hunting): handle_auth_done global_id 22133 payload 824 2024-09-11T05:10:06.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877a2f1640 1 -- 172.21.15.138:0/2883319791 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8774140d50 con 0x7f8774140970 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f876401c460 con 0x7f877409e6e0 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 1 --2- 172.21.15.138:0/2883319791 >> v2:172.21.15.190:6800/1562782189 conn(0x7f875803ced0 0x7f875803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(87..87 src has 1..87) ==== 7615+0+0 (secure 0 0 0) 0x7f87640500a0 con 0x7f877409e6e0 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 10 client.22130.objecter ms_dispatch 0x7f8774000ba0 osd_map(87..87 src has 1..87) 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 3 client.22130.objecter handle_osd_map got epochs [87,87] > 0 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 3 client.22130.objecter handle_osd_map decoding full epoch 87 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f876affd640 20 client.22130.objecter dump_active .. 0 homeless 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877aaf2640 1 --2- 172.21.15.138:0/2883319791 >> v2:172.21.15.190:6800/1562782189 conn(0x7f875803ced0 0x7f875803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.079+0000 7f877aaf2640 10 monclient: get_auth_request con 0x7f875803ced0 auth_method 0 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f877a2f1640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8774140970 msgr2=0x7f87741373b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f877a2f1640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8774140970 0x7f87741373b0 secure :-1 s=READY pgs=5607 cs=0 l=1 rev1=1 crypto rx=0x7f877000c6f0 tx=0x7f877000cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 10 client.22130.objecter ms_handle_connect 0x7f8774140970 2024-09-11T05:10:06.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 10 client.22130.objecter resend_mon_ops 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770003dd0 con 0x7f8774140970 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 10 monclient: dump: 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.097 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8770003dd0 con 0x7f8774140970 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8774140970 msgr2=0x7f87741373b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8774140970 0x7f87741373b0 secure :-1 s=READY pgs=5607 cs=0 l=1 rev1=1 crypto rx=0x7f877000c6f0 tx=0x7f877000cbb0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8774140970 msgr2=0x7f87741373b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8774140970 0x7f87741373b0 secure :-1 s=CLOSED pgs=5607 cs=0 l=1 rev1=1 crypto rx=0x7f877000c6f0 tx=0x7f877000cbb0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f877aaf2640 1 --2- 172.21.15.138:0/2883319791 >> v2:172.21.15.190:6800/1562782189 conn(0x7f875803ced0 0x7f875803f390 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f877406daa0 tx=0x7f876c0a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.083+0000 7f876affd640 10 client.22130.objecter ms_handle_connect 0x7f875803ced0 2024-09-11T05:10:06.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877c555640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:10:06.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877c555640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f87741378f0 0x7f87740a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877c555640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8774137d00 con 0x7f87741378f0 2024-09-11T05:10:06.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f87741378f0 0x7f87740a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 10 monclient: get_auth_request con 0x7f87741378f0 auth_method 0 2024-09-11T05:10:06.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 10 monclient(hunting): handle_auth_done global_id 22136 payload 824 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.087+0000 7f877a2f1640 1 -- 172.21.15.138:0/2883319791 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8774135450 con 0x7f87741378f0 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877a2f1640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f87741378f0 msgr2=0x7f87740a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877a2f1640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f87741378f0 0x7f87740a4550 secure :-1 s=READY pgs=5608 cs=0 l=1 rev1=1 crypto rx=0x7f877000c6f0 tx=0x7f87700059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 10 client.22130.objecter ms_handle_connect 0x7f87741378f0 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 10 client.22130.objecter resend_mon_ops 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877001c110 con 0x7f87741378f0 2024-09-11T05:10:06.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 10 monclient: dump: 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.102 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f877001c110 con 0x7f87741378f0 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f87741378f0 msgr2=0x7f87740a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f87741378f0 0x7f87740a4550 secure :-1 s=READY pgs=5608 cs=0 l=1 rev1=1 crypto rx=0x7f877000c6f0 tx=0x7f87700059e0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f87741378f0 msgr2=0x7f87740a4550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f876affd640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f87741378f0 0x7f87740a4550 secure :-1 s=CLOSED pgs=5608 cs=0 l=1 rev1=1 crypto rx=0x7f877000c6f0 tx=0x7f87700059e0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:10:06.103 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 16, 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 164864, 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:10:06.104 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:10:06.105 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:10:06.106 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:10:06.107 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:10:06.108 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:10:06.109 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:10:06.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:10:06.111 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:10:06.112 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.113 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:10:06.114 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:10:06.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.116 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:10:06.116 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:10:06.116 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:10:06.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.116 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.116 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:10:06.116 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 10 librados: watch_flush enter 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 10 librados: watch_flush exit 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 20 client.22130.objecter shutdown clearing up homeless session... 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 10 client.22130.objecter successfully canceled tick 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 >> v2:172.21.15.190:6800/1562782189 conn(0x7f875803ced0 msgr2=0x7f875803f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 --2- 172.21.15.138:0/2883319791 >> v2:172.21.15.190:6800/1562782189 conn(0x7f875803ced0 0x7f875803f390 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f877406daa0 tx=0x7f876c0a7070 comp rx=0 tx=0).stop 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 10 monclient: shutdown 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:10:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409e6e0 msgr2=0x7f8774140430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409e6e0 0x7f8774140430 secure :-1 s=READY pgs=5606 cs=0 l=1 rev1=1 crypto rx=0x7f876400c450 tx=0x7f876400c910 comp rx=0 tx=0).stop 2024-09-11T05:10:06.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877aaf2640 1 -- 172.21.15.138:0/2883319791 reap_dead start 2024-09-11T05:10:06.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 shutdown_connections 2024-09-11T05:10:06.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 --2- 172.21.15.138:0/2883319791 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f877409e6e0 0x7f8774140430 unknown :-1 s=CLOSED pgs=5606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 >> 172.21.15.138:0/2883319791 conn(0x7f87740ab830 msgr2=0x7f87740a9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:10:06.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 shutdown_connections 2024-09-11T05:10:06.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 -- 172.21.15.138:0/2883319791 wait complete. 2024-09-11T05:10:06.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.091+0000 7f877c555640 1 librados: shutdown 2024-09-11T05:10:06.119 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:10:06.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T05:10:06.120 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:10:06.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient: get_monmap_and_config 2024-09-11T05:10:06.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient: build_initial_monmap 2024-09-11T05:10:06.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient: monmap: 2024-09-11T05:10:06.233 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:10:06.233 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.233 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:10:06.234347+0000 2024-09-11T05:10:06.233 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:10:06.234347+0000 2024-09-11T05:10:06.233 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:10:06.234 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.234 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:10:06.234 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:10:06.234 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:10:06.234 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:10:06.234 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:10:06.234 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 1 Processor -- start 2024-09-11T05:10:06.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 1 -- start start 2024-09-11T05:10:06.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient: init 2024-09-11T05:10:06.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient: _reopen_session rank -1 2024-09-11T05:10:06.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-11T05:10:06.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c0a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient(hunting): picked mon.noname-b con 0x7f690c09da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f690c0a1e60 addr v1:172.21.15.190:6789/0 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f690c0a22a0 0x7f690c136910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient(hunting): picked mon.noname-c con 0x7f690c0a22a0 addr v2:172.21.15.190:3300/0 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f690c064590 con 0x7f690c0a1e60 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f690c09de00 con 0x7f690c0a22a0 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f690c0a2c70 con 0x7f690c09da20 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient(hunting): _renew_subs 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f6912190640 10 monclient(hunting): authenticate will time out at 1089.670967s 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c0a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c0a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:45856/0 (socket says 172.21.15.138:45856) 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 1 -- 172.21.15.138:0/2325241499 learned_addr learned my addr 172.21.15.138:0/2325241499 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient(hunting): get_auth_request con 0x7f690c09da20 auth_method 0 2024-09-11T05:10:06.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690bfff640 1 --2- 172.21.15.138:0/2325241499 >> v2:172.21.15.190:3300/0 conn(0x7f690c0a22a0 0x7f690c136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690bfff640 10 monclient(hunting): get_auth_request con 0x7f690c0a22a0 auth_method 0 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient(hunting): handle_auth_done global_id 22139 payload 274 2024-09-11T05:10:06.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 1 -- 172.21.15.138:0/2325241499 >> v2:172.21.15.190:3300/0 conn(0x7f690c0a22a0 msgr2=0x7f690c136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:10:06.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 1 --2- 172.21.15.138:0/2325241499 >> v2:172.21.15.190:3300/0 conn(0x7f690c0a22a0 0x7f690c136910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 1 -- 172.21.15.138:0/2325241499 >> v1:172.21.15.190:6789/0 conn(0x7f690c0a1e60 legacy=0x7f690c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:10:06.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:10:06.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 1 monclient: found mon.noname-b 2024-09-11T05:10:06.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:10:06.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:10:06.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.231+0000 7f690b7fe640 1 -- 172.21.15.138:0/2325241499 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f690c0a2f50 con 0x7f690c09da20 2024-09-11T05:10:06.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 -- 172.21.15.138:0/2325241499 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 msgr2=0x7f690c0a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:06.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 --2- 172.21.15.138:0/2325241499 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c0a1920 secure :-1 s=READY pgs=5609 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009870 tx=0x7f68fc01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1037f4eb5281a0dd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 1 -- 172.21.15.138:0/2325241499 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc0b2070 con 0x7f690c09da20 2024-09-11T05:10:06.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:10:06.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: dump: 2024-09-11T05:10:06.244 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: _finish_auth 0 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:10:06.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 1 -- 172.21.15.138:0/2325241499 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68fc0a9070 con 0x7f690c09da20 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 1 -- 172.21.15.138:0/2325241499 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc0ad880 con 0x7f690c09da20 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 5 monclient: authenticate success, global_id 22139 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient: get_monmap_and_config success 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 4 set_mon_vals no callback set 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690a7fc640 10 monclient: dump: 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.246 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.247 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.247 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.247 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.247 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.247 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient: shutdown 2024-09-11T05:10:06.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:10:06.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- 172.21.15.138:0/2325241499 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 msgr2=0x7f690c0a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 --2- 172.21.15.138:0/2325241499 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c0a1920 secure :-1 s=READY pgs=5609 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009870 tx=0x7f68fc01c420 comp rx=0 tx=0).stop 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- 172.21.15.138:0/2325241499 shutdown_connections 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 --2- 172.21.15.138:0/2325241499 >> v2:172.21.15.190:3300/0 conn(0x7f690c0a22a0 0x7f690c136910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 --2- 172.21.15.138:0/2325241499 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c0a1920 unknown :-1 s=CLOSED pgs=5609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- 172.21.15.138:0/2325241499 >> 172.21.15.138:0/2325241499 conn(0x7f690c063640 msgr2=0x7f690c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- 172.21.15.138:0/2325241499 shutdown_connections 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- 172.21.15.138:0/2325241499 wait complete. 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient: build_initial_monmap 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient: monmap: 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:10:06.248 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 librados: starting msgr at 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 librados: starting objecter 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 Processor -- start 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- start start 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 librados: setting wanted keys 2024-09-11T05:10:06.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 librados: calling monclient init 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient: init 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient: _reopen_session rank -1 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient(hunting): picked mon.noname-c con 0x7f690c09da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c0a1e60 0x7f690c09e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient(hunting): picked mon.noname-b con 0x7f690c0a1e60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f690c0a22a0 0x7f690c09e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient(hunting): picked mon.noname-a con 0x7f690c0a22a0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f690c140ee0 con 0x7f690c0a22a0 2024-09-11T05:10:06.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f690c140d60 con 0x7f690c0a1e60 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f690c141060 con 0x7f690c09da20 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient(hunting): _renew_subs 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f6912190640 10 monclient(hunting): authenticate will time out at 1089.678463s 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690bfff640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f690c0a22a0 msgr2=0x7f690c09e990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f690c0a22a0 0x7f690c09e990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:45862/0 (socket says 172.21.15.138:45862) 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 -- 172.21.15.138:0/1120871346 learned_addr learned my addr 172.21.15.138:0/1120871346 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient(hunting): get_auth_request con 0x7f690c09da20 auth_method 0 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient(hunting): handle_auth_done global_id 22142 payload 824 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c0a1e60 msgr2=0x7f690c09e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c0a1e60 0x7f690c09e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f690c0a22a0 msgr2=0x7f690c09e990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f690c0a22a0 0x7f690c09e990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 monclient: found mon.noname-c 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:10:06.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.239+0000 7f690b7fe640 1 -- 172.21.15.138:0/1120871346 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f690c13cff0 con 0x7f690c09da20 2024-09-11T05:10:06.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690b7fe640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 msgr2=0x7f690c073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:06.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690b7fe640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c073c10 secure :-1 s=READY pgs=5610 cs=0 l=1 rev1=1 crypto rx=0x7f68fc0b0040 tx=0x7f68fc01ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 client.?.objecter ms_handle_connect 0x7f690c09da20 2024-09-11T05:10:06.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:10:06.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc0bc020 con 0x7f690c09da20 2024-09-11T05:10:06.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:10:06.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: dump: 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.257 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: _finish_auth 0 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: _check_auth_tickets 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68fc0b8620 con 0x7f690c09da20 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc0b2040 con 0x7f690c09da20 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 monclient: dump: 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.258 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 5 monclient: authenticate success, global_id 22142 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 10 monclient: _renew_subs 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f690c136b80 con 0x7f690c09da20 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f69097fa640 4 set_mon_vals no callback set 2024-09-11T05:10:06.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 10 client.22142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 10 monclient: _renew_subs 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f690c09fe90 con 0x7f690c09da20 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 1 librados: init done 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c09eed0 0x7f690c12d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f690c09f2b0 con 0x7f690c09eed0 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f68fc0b5030 con 0x7f690c09da20 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c09eed0 0x7f690c12d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 10 monclient: get_auth_request con 0x7f690c09eed0 auth_method 0 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 1 --2- 172.21.15.138:0/1120871346 >> v2:172.21.15.190:6800/1562782189 conn(0x7f68f803ce80 0x7f68f803f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(87..87 src has 1..87) ==== 7615+0+0 (secure 0 0 0) 0x7f68fc0ea5c0 con 0x7f690c09da20 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 10 client.22142.objecter ms_dispatch 0x7f690c000ba0 osd_map(87..87 src has 1..87) 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 3 client.22142.objecter handle_osd_map got epochs [87,87] > 0 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 3 client.22142.objecter handle_osd_map decoding full epoch 87 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f68f3fff640 20 client.22142.objecter dump_active .. 0 homeless 2024-09-11T05:10:06.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690bfff640 1 --2- 172.21.15.138:0/1120871346 >> v2:172.21.15.190:6800/1562782189 conn(0x7f68f803ce80 0x7f68f803f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690bfff640 10 monclient: get_auth_request con 0x7f68f803ce80 auth_method 0 2024-09-11T05:10:06.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 10 monclient(hunting): handle_auth_done global_id 20236 payload 824 2024-09-11T05:10:06.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.243+0000 7f690affd640 1 -- 172.21.15.138:0/1120871346 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f690c0a1920 con 0x7f690c09eed0 2024-09-11T05:10:06.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 10 client.22142.objecter ms_handle_connect 0x7f690c09eed0 2024-09-11T05:10:06.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f690affd640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c09eed0 msgr2=0x7f690c12d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:06.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 10 client.22142.objecter resend_mon_ops 2024-09-11T05:10:06.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f690affd640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c09eed0 0x7f690c12d880 secure :-1 s=READY pgs=5602 cs=0 l=1 rev1=1 crypto rx=0x7f690c09e050 tx=0x7f690000c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6900018070 con 0x7f690c09eed0 2024-09-11T05:10:06.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:10:06.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 10 monclient: dump: 2024-09-11T05:10:06.263 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.263 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.264 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.264 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.264 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.264 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.264 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.265 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.265 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.265 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f690bfff640 1 --2- 172.21.15.138:0/1120871346 >> v2:172.21.15.190:6800/1562782189 conn(0x7f68f803ce80 0x7f68f803f340 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f6904009660 tx=0x7f69040098b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 10 client.22142.objecter ms_handle_connect 0x7f68f803ce80 2024-09-11T05:10:06.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6900004800 con 0x7f690c09eed0 2024-09-11T05:10:06.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:10:06.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:10:06.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c09eed0 msgr2=0x7f690c12d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c09eed0 0x7f690c12d880 secure :-1 s=READY pgs=5602 cs=0 l=1 rev1=1 crypto rx=0x7f690c09e050 tx=0x7f690000c7a0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c09eed0 msgr2=0x7f690c12d880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:10:06.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.247+0000 7f68f3fff640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c09eed0 0x7f690c12d880 secure :-1 s=CLOSED pgs=5602 cs=0 l=1 rev1=1 crypto rx=0x7f690c09e050 tx=0x7f690000c7a0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f6912190640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:10:06.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f6912190640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c12ddc0 0x7f690c12e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:10:06.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f6912190640 10 monclient(hunting): start opening mon connection 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f690c13bad0 con 0x7f690c12ddc0 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c12ddc0 0x7f690c12e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 10 monclient: get_auth_request con 0x7f690c12ddc0 auth_method 0 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:10:06.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 10 monclient(hunting): handle_auth_done global_id 20239 payload 824 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.255+0000 7f690affd640 1 -- 172.21.15.138:0/1120871346 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f690c06cca0 con 0x7f690c12ddc0 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f690affd640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c12ddc0 msgr2=0x7f690c12e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 10 client.22142.objecter ms_handle_connect 0x7f690c12ddc0 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 10 client.22142.objecter resend_mon_ops 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f690affd640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c12ddc0 0x7f690c12e1d0 secure :-1 s=READY pgs=5603 cs=0 l=1 rev1=1 crypto rx=0x7f6900015000 tx=0x7f6900003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6900018030 con 0x7f690c12ddc0 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 10 monclient: dump: 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.272 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f690001aa40 con 0x7f690c12ddc0 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c12ddc0 msgr2=0x7f690c12e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c12ddc0 0x7f690c12e1d0 secure :-1 s=READY pgs=5603 cs=0 l=1 rev1=1 crypto rx=0x7f6900015000 tx=0x7f6900003770 comp rx=0 tx=0).stop 2024-09-11T05:10:06.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c12ddc0 msgr2=0x7f690c12e1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f68f3fff640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f690c12ddc0 0x7f690c12e1d0 secure :-1 s=CLOSED pgs=5603 cs=0 l=1 rev1=1 crypto rx=0x7f6900015000 tx=0x7f6900003770 comp rx=0 tx=0).stop 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 16, 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 164993, 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:10:06.274 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:10:06.275 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:10:06.276 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:10:06.277 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:10:06.278 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:10:06.279 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:10:06.280 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.281 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:10:06.282 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.283 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:10:06.284 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:10:06.285 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.285 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:10:06.285 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:10:06.285 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:10:06.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.285 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:10:06.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.287 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 9 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:10:06.288 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:10:06.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:10:06.289 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:10:06.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:10:06.289 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:10:06.289 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 10 librados: watch_flush enter 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 10 librados: watch_flush exit 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 20 client.22142.objecter shutdown clearing up homeless session... 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 10 client.22142.objecter successfully canceled tick 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 >> v2:172.21.15.190:6800/1562782189 conn(0x7f68f803ce80 msgr2=0x7f68f803f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 --2- 172.21.15.138:0/1120871346 >> v2:172.21.15.190:6800/1562782189 conn(0x7f68f803ce80 0x7f68f803f340 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f6904009660 tx=0x7f69040098b0 comp rx=0 tx=0).stop 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 10 monclient: shutdown 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:10:06.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 msgr2=0x7f690c073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:10:06.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 --2- 172.21.15.138:0/1120871346 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f690c09da20 0x7f690c073c10 secure :-1 s=READY pgs=5610 cs=0 l=1 rev1=1 crypto rx=0x7f68fc0b0040 tx=0x7f68fc01ac60 comp rx=0 tx=0).stop 2024-09-11T05:10:06.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f690bfff640 1 -- 172.21.15.138:0/1120871346 reap_dead start 2024-09-11T05:10:06.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f690bfff640 1 -- 172.21.15.138:0/1120871346 reap_dead start 2024-09-11T05:10:06.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 shutdown_connections 2024-09-11T05:10:06.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 >> 172.21.15.138:0/1120871346 conn(0x7f690c063640 msgr2=0x7f690c135fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:10:06.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 shutdown_connections 2024-09-11T05:10:06.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 -- 172.21.15.138:0/1120871346 wait complete. 2024-09-11T05:10:06.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:10:06.259+0000 7f6912190640 1 librados: shutdown 2024-09-11T05:10:06.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:10:06.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:10:09.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-11T05:10:09.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-11T05:10:09.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-11T05:10:09.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-11T05:10:09.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: seed 28166 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (28 ms) 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (193 ms) 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (24 ms) 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (5 ms) 2024-09-11T05:10:09.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (18 ms) 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (3 ms) 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (11 ms) 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (1 ms) 2024-09-11T05:10:09.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (8 ms) 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (18 ms) 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (4351 ms) 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (581 ms) 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: done waiting, doing copies 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-11T05:10:09.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: done waiting 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60688 ms) 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (13 ms) 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (16013 ms) 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-11T05:10:09.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 22 tests from LibRadosMis api_snapshots_pp: Running main() from gmock_main.cc 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: seed 28280 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2026 ms) 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-11T05:10:10.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2003 ms) 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2026 ms) 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2007 ms) 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4005 ms) 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (12067 ms total) 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4116 ms) 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3918 ms) 2024-09-11T05:10:10.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6172 ms) 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3985 ms) 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (1954 ms) 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi190-28280-7 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-11T05:10:10.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17910 ms) 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38055 ms total) 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (23 ms) 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5270 ms) 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5293 ms total) 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (17189 ms) 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-11T05:10:10.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2009 ms) 2024-09-11T05:10:10.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-11T05:10:10.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2002 ms) 2024-09-11T05:10:25.372 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots api_aio: Running main() from gmock_main.cc 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] Global test environment set-up. 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2269 ms) 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3014 ms) 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3017 ms) 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3485 ms) 2024-09-11T05:10:25.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3536 ms) 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2438 ms) 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3999 ms) 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3045 ms) 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2964 ms) 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3011 ms) 2024-09-11T05:10:25.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3189 ms) 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2799 ms) 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3055 ms) 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2979 ms) 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.Flush (2962 ms) 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3045 ms) 2024-09-11T05:10:25.375 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3000 ms) 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3022 ms) 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2950 ms) 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3228 ms) 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3002 ms) 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (17188 ms) 2024-09-11T05:10:25.376 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3006 ms) 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2584 ms) 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3015 ms) 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2980 ms) 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] 26 tests from LibRadosAio (92782 ms total) 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-11T05:10:25.377 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-11T05:10:33.390 INFO:tasks.workunit.client.0.smithi190.stdout:_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-11T05:10:33.390 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1570 ms) 2024-09-11T05:10:33.390 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (22770 ms total) 2024-09-11T05:10:33.390 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T05:10:33.390 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-11T05:10:33.390 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-11T05:10:33.390 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4053 ms) 2024-09-11T05:10:33.390 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3938 ms) 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4027 ms) 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12018 ms total) 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (107701 ms total) 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-11T05:10:33.391 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout:cPP (81969 ms total) 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (186 ms) 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (186 ms total) 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-11T05:10:35.397 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (68 ms) 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (9 ms) 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (77 ms total) 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (61 ms) 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (10 ms) 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (71 ms total) 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T05:10:35.398 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (63 ms) 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (216 ms) 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (279 ms total) 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1046 ms) 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1046 ms total) 2024-09-11T05:10:35.399 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T05:10:35.400 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-11T05:10:35.400 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (109731 ms total) 2024-09-11T05:10:35.400 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-11T05:10:40.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-11T05:10:40.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-11T05:10:40.421 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3260 ms) 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19520 ms) 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6016 ms) 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3037 ms) 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3125 ms) 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2805 ms) 2024-09-11T05:10:40.422 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2989 ms) 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3042 ms) 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2976 ms) 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3036 ms) 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3009 ms) 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2956 ms) 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3199 ms) 2024-09-11T05:10:40.423 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3045 ms) 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (17176 ms) 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3002 ms) 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3020 ms) 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2573 ms) 2024-09-11T05:10:40.424 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2990 ms) 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3013 ms) 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2991 ms) 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3013 ms) 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2993 ms) 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3002 ms) 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3014 ms) 2024-09-11T05:10:40.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-11T05:10:40.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3023 ms) 2024-09-11T05:10:40.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-11T05:11:30.773 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAi api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=9 expected=9 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=12 expected=12 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:566253c9:::13:head 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-09-11T05:11:30.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=13 expected=13 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=4 expected=4 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=3 expected=3 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=6 expected=6 2024-09-11T05:11:30.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=8 expected=8 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=5 expected=5 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (375 ms) 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (121823 ms) 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-09-11T05:11:30.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (36114 ms) 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 7 tests from LibRadosList (158788 ms total) 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1088 ms) 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo1,foo2,foo3 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo1 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo2 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo3 2024-09-11T05:11:30.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo1,foo4,foo5 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo4 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo5 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo1 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo6,foo7 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo7 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo6 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo4 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo5 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns2:foo7 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns2:foo6 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo1 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo1 2024-09-11T05:11:30.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo2 2024-09-11T05:11:30.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo3 2024-09-11T05:11:30.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (109 ms) 2024-09-11T05:11:30.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-11T05:11:30.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 1 0 2024-09-11T05:11:30.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 10 0 2024-09-11T05:11:30.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 13 0 2024-09-11T05:11:36.293 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T05:11:36.294 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T05:11:36.294 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T05:11:36.297 INFO:tasks.ceph.mon.a:Started 2024-09-11T05:11:36.297 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:11:36.297 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient: get_monmap_and_config 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient: build_initial_monmap 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient: monmap: 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:11:36.422528+0000 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:11:36.422528+0000 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:11:36.421 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:11:36.422 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:11:36.422 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:11:36.422 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 1 Processor -- start 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 1 -- start start 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient: init 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f44b40a5b40 0x7f44b40a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient(hunting): picked mon.noname-c con 0x7f44b40a5b40 addr v2:172.21.15.190:3300/0 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44b40a7130 addr v1:172.21.15.138:6789/0 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b409d840 0x7f44b40a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient(hunting): picked mon.noname-b con 0x7f44b409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44b409d490 con 0x7f44b40a7130 2024-09-11T05:11:36.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f44b409e110 con 0x7f44b40a5b40 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f44b40a69a0 con 0x7f44b409d840 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44bb827640 10 monclient(hunting): authenticate will time out at 1179.858696s 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b409d840 0x7f44b40a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b409d840 0x7f44b40a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:52154/0 (socket says 172.21.15.138:52154) 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 1 -- 172.21.15.138:0/1025996262 learned_addr learned my addr 172.21.15.138:0/1025996262 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient(hunting): get_auth_request con 0x7f44b409d840 auth_method 0 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:36.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient(hunting): handle_auth_done global_id 22364 payload 274 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 1 -- 172.21.15.138:0/1025996262 >> v2:172.21.15.190:3300/0 conn(0x7f44b40a5b40 msgr2=0x7f44b40a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 1 --2- 172.21.15.138:0/1025996262 >> v2:172.21.15.190:3300/0 conn(0x7f44b40a5b40 0x7f44b40a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 1 -- 172.21.15.138:0/1025996262 >> v1:172.21.15.138:6789/0 conn(0x7f44b40a7130 legacy=0x7f44b4138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient: _finish_hunting 0 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 1 monclient: found mon.noname-b 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:11:36.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.417+0000 7f44b9dc4640 1 -- 172.21.15.138:0/1025996262 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b4070580 con 0x7f44b409d840 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44b9dc4640 1 -- 172.21.15.138:0/1025996262 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b409d840 msgr2=0x7f44b40a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44b9dc4640 1 --2- 172.21.15.138:0/1025996262 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b409d840 0x7f44b40a6460 secure :-1 s=READY pgs=5665 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009e50 tx=0x7f44ac01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=909dcf320ea6da56 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 1 -- 172.21.15.138:0/1025996262 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0a5560 con 0x7f44b409d840 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: dump: 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:36.428 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: _finish_auth 0 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: _check_auth_tickets 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 1 -- 172.21.15.138:0/1025996262 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44ac0a5700 con 0x7f44b409d840 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44bb827640 5 monclient: authenticate success, global_id 22364 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44bb827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44bb827640 10 monclient: get_monmap_and_config success 2024-09-11T05:11:36.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 1 -- 172.21.15.138:0/1025996262 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0a5ad0 con 0x7f44b409d840 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44bb827640 4 set_mon_vals no callback set 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.421+0000 7f44abfff640 10 monclient: dump: 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:36.430 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient: shutdown 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- 172.21.15.138:0/1025996262 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b409d840 msgr2=0x7f44b40a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 --2- 172.21.15.138:0/1025996262 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b409d840 0x7f44b40a6460 secure :-1 s=READY pgs=5665 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009e50 tx=0x7f44ac01c8f0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- 172.21.15.138:0/1025996262 shutdown_connections 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 --2- 172.21.15.138:0/1025996262 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b409d840 0x7f44b40a6460 unknown :-1 s=CLOSED pgs=5665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 --2- 172.21.15.138:0/1025996262 >> v2:172.21.15.190:3300/0 conn(0x7f44b40a5b40 0x7f44b40a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- 172.21.15.138:0/1025996262 >> 172.21.15.138:0/1025996262 conn(0x7f44b40aba10 msgr2=0x7f44b40a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- 172.21.15.138:0/1025996262 shutdown_connections 2024-09-11T05:11:36.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- 172.21.15.138:0/1025996262 wait complete. 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient: build_initial_monmap 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient: monmap: 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:11:36.435 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 librados: starting msgr at 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 librados: starting objecter 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 Processor -- start 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- start start 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 librados: setting wanted keys 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 librados: calling monclient init 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient: init 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 0x7f44b41d0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient(hunting): picked mon.noname-a con 0x7f44b409d840 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b40a5b40 0x7f44b41d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:36.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient(hunting): picked mon.noname-b con 0x7f44b40a5b40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b40a7130 0x7f44b413cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient(hunting): picked mon.noname-c con 0x7f44b40a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f44b4142f60 con 0x7f44b409d840 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f44b4142de0 con 0x7f44b40a5b40 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f44b41430e0 con 0x7f44b40a7130 2024-09-11T05:11:36.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44bb827640 10 monclient(hunting): authenticate will time out at 1179.865779s 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 0x7f44b41d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b40a7130 0x7f44b413cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b40a7130 0x7f44b413cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:52168/0 (socket says 172.21.15.138:52168) 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 -- 172.21.15.138:0/2880598307 learned_addr learned my addr 172.21.15.138:0/2880598307 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient(hunting): get_auth_request con 0x7f44b40a7130 auth_method 0 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 10 monclient(hunting): get_auth_request con 0x7f44b409d840 auth_method 0 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:36.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 1 -- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 msgr2=0x7f44b41d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 1 -- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 msgr2=0x7f44b41d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 0x7f44b41d0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b95c3640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 0x7f44b41d0600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient(hunting): handle_auth_done global_id 22367 payload 824 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 -- 172.21.15.138:0/2880598307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b40a5b40 msgr2=0x7f44b41d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b40a5b40 0x7f44b41d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 -- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 msgr2=0x7f44b41d0600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 0x7f44b41d0600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient: _finish_hunting 0 2024-09-11T05:11:36.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 monclient: found mon.noname-c 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.425+0000 7f44b9dc4640 1 -- 172.21.15.138:0/2880598307 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b4122590 con 0x7f44b40a7130 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44b9dc4640 1 -- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b40a7130 msgr2=0x7f44b413cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44b9dc4640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b40a7130 0x7f44b413cb40 secure :-1 s=READY pgs=5666 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0af040 tx=0x7f44ac019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 client.?.objecter ms_handle_connect 0x7f44b40a7130 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 1 -- 172.21.15.138:0/2880598307 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0bc020 con 0x7f44b40a7130 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: dump: 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:36.440 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: _finish_auth 0 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 5 monclient: authenticate success, global_id 22367 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 10 monclient: _renew_subs 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 1 -- 172.21.15.138:0/2880598307 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44ac0b8880 con 0x7f44b40a7130 2024-09-11T05:11:36.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b413e600 con 0x7f44b40a7130 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 1 -- 172.21.15.138:0/2880598307 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ac0a5560 con 0x7f44b40a7130 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44a9ffb640 10 monclient: dump: 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:36.442 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44ab7fe640 4 set_mon_vals no callback set 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 10 client.22367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 10 monclient: _renew_subs 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b413f8f0 con 0x7f44b40a7130 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 1 librados: init done 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:36.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.429+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b41220b0 con 0x7f44b40a7130 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 1 -- 172.21.15.138:0/2880598307 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f44ac0b8c50 con 0x7f44b40a7130 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 1 --2- 172.21.15.138:0/2880598307 >> v2:172.21.15.190:6800/1562782189 conn(0x7f449803cea0 0x7f449803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 1 -- 172.21.15.138:0/2880598307 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(178..178 src has 1..178) ==== 5286+0+0 (secure 0 0 0) 0x7f44ac0eb050 con 0x7f44b40a7130 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 10 client.22367.objecter ms_dispatch 0x7f44b4000ba0 osd_map(178..178 src has 1..178) 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 3 client.22367.objecter handle_osd_map got epochs [178,178] > 0 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 3 client.22367.objecter handle_osd_map decoding full epoch 178 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 20 client.22367.objecter dump_active .. 0 homeless 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 1 -- 172.21.15.138:0/2880598307 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f44b41220b0 con 0x7f44b40a7130 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:36.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.441+0000 7f44a9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:11:36.447 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 7 0 2024-09-11T05:11:36.447 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 14 0 2024-09-11T05:11:36.447 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 0 0 2024-09-11T05:11:36.447 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 15 0 2024-09-11T05:11:36.447 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 11 0 2024-09-11T05:11:36.447 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 5 0 2024-09-11T05:11:36.447 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 8 0 2024-09-11T05:11:36.447 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 6 0 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 3 0 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 4 0 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 12 0 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 9 0 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2 0 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (150 ms) 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 3 tests from LibRadosListEC (1347 ms total) 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3040 ms) 2024-09-11T05:11:36.448 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 1 test from LibRadosListNP (3040 ms total) 2024-09-11T05:11:36.449 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2024-09-11T05:11:36.449 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] Global test environment tear-down 2024-09-11T05:11:36.449 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [==========] 11 tests from 3 test suites ran. (170821 ms total) 2024-09-11T05:11:36.449 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ PASSED ] 11 tests. 2024-09-11T05:11:36.451 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.449+0000 7f44b95c3640 1 --2- 172.21.15.138:0/2880598307 >> v2:172.21.15.190:6800/1562782189 conn(0x7f449803cea0 0x7f449803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:36.451 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.449+0000 7f44b95c3640 10 monclient: get_auth_request con 0x7f449803cea0 auth_method 0 2024-09-11T05:11:36.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.449+0000 7f44a9ffb640 10 client.22367.objecter ms_handle_connect 0x7f449803cea0 2024-09-11T05:11:36.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.449+0000 7f44b95c3640 1 --2- 172.21.15.138:0/2880598307 >> v2:172.21.15.190:6800/1562782189 conn(0x7f449803cea0 0x7f449803f360 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f44b0006ed0 tx=0x7f44b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:36.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.681+0000 7f44bb827640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:11:36.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.681+0000 7f44bb827640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:36.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.681+0000 7f44bb827640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:36.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.681+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f44b4138900 con 0x7f44b40a7130 2024-09-11T05:11:36.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44a9ffb640 1 -- 172.21.15.138:0/2880598307 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f44ac0c6030 con 0x7f44b40a7130 2024-09-11T05:11:36.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:36.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44a9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:11:36.689 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":92,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:11:36.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 10 librados: watch_flush enter 2024-09-11T05:11:36.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 10 librados: watch_flush exit 2024-09-11T05:11:36.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 20 client.22367.objecter shutdown clearing up homeless session... 2024-09-11T05:11:36.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 10 client.22367.objecter successfully canceled tick 2024-09-11T05:11:36.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 >> v2:172.21.15.190:6800/1562782189 conn(0x7f449803cea0 msgr2=0x7f449803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:36.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 --2- 172.21.15.138:0/2880598307 >> v2:172.21.15.190:6800/1562782189 conn(0x7f449803cea0 0x7f449803f360 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f44b0006ed0 tx=0x7f44b0008040 comp rx=0 tx=0).stop 2024-09-11T05:11:36.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 10 monclient: shutdown 2024-09-11T05:11:36.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:36.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b40a7130 msgr2=0x7f44b413cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:36.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b40a7130 0x7f44b413cb40 secure :-1 s=READY pgs=5666 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0af040 tx=0x7f44ac019690 comp rx=0 tx=0).stop 2024-09-11T05:11:36.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 shutdown_connections 2024-09-11T05:11:36.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 --2- 172.21.15.138:0/2880598307 >> v2:172.21.15.190:6800/1562782189 conn(0x7f449803cea0 0x7f449803f360 unknown :-1 s=CLOSED pgs=3343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f44b40a7130 0x7f44b413cb40 unknown :-1 s=CLOSED pgs=5666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44b40a5b40 0x7f44b41d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 --2- 172.21.15.138:0/2880598307 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f44b409d840 0x7f44b41d0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:36.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 >> 172.21.15.138:0/2880598307 conn(0x7f44b40aba10 msgr2=0x7f44b40a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:36.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 shutdown_connections 2024-09-11T05:11:36.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 -- 172.21.15.138:0/2880598307 wait complete. 2024-09-11T05:11:36.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:36.685+0000 7f44bb827640 1 librados: shutdown 2024-09-11T05:11:39.708 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:11:39.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.833+0000 7f5858c7e640 10 monclient: get_monmap_and_config 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.833+0000 7f5858c7e640 10 monclient: build_initial_monmap 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.833+0000 7f5858c7e640 10 monclient: monmap: 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:11:39.839333+0000 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:11:39.839333+0000 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:11:39.838 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:11:39.839 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:11:39.839 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:39.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.833+0000 7f5858c7e640 1 Processor -- start 2024-09-11T05:11:39.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.833+0000 7f5858c7e640 1 -- start start 2024-09-11T05:11:39.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.833+0000 7f5858c7e640 10 monclient: init 2024-09-11T05:11:39.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:39.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-11T05:11:39.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4030 0x7f58540a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:39.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient(hunting): picked mon.noname-b con 0x7f58540a4030 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:39.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58540a4410 addr v1:172.21.15.190:6789/0 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 0x7f585413fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient(hunting): picked mon.noname-a con 0x7f585413da40 addr v2:172.21.15.138:3300/0 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58540a2850 con 0x7f58540a4410 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f58540a30f0 con 0x7f585413da40 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f58540a2b30 con 0x7f58540a4030 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5858c7e640 10 monclient(hunting): authenticate will time out at 1183.276262s 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4030 0x7f58540a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 0x7f585413fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4030 0x7f58540a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48406/0 (socket says 172.21.15.138:48406) 2024-09-11T05:11:39.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 0x7f585413fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:41822/0 (socket says 172.21.15.138:41822) 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 1 -- 172.21.15.138:0/699230522 learned_addr learned my addr 172.21.15.138:0/699230522 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient(hunting): get_auth_request con 0x7f58540a4030 auth_method 0 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 10 monclient(hunting): get_auth_request con 0x7f585413da40 auth_method 0 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 1 -- 172.21.15.138:0/699230522 >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 msgr2=0x7f585413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 1 -- 172.21.15.138:0/699230522 >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 msgr2=0x7f585413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 1 --2- 172.21.15.138:0/699230522 >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 0x7f585413fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:39.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f5852d9d640 1 --2- 172.21.15.138:0/699230522 >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 0x7f585413fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient(hunting): handle_auth_done global_id 22373 payload 274 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 1 -- 172.21.15.138:0/699230522 >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 msgr2=0x7f585413fe30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 1 --2- 172.21.15.138:0/699230522 >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 0x7f585413fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 1 -- 172.21.15.138:0/699230522 >> v1:172.21.15.190:6789/0 conn(0x7f58540a4410 legacy=0x7f58540a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient: _finish_hunting 0 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 1 monclient: found mon.noname-b 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:11:39.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.837+0000 7f585259c640 1 -- 172.21.15.138:0/699230522 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58540a2e10 con 0x7f58540a4030 2024-09-11T05:11:39.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f585259c640 1 -- 172.21.15.138:0/699230522 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4030 msgr2=0x7f58540a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f585259c640 1 --2- 172.21.15.138:0/699230522 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4030 0x7f58540a3ab0 secure :-1 s=READY pgs=5667 cs=0 l=1 rev1=1 crypto rx=0x7f5844009870 tx=0x7f584401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2627606badb98c02 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 1 -- 172.21.15.138:0/699230522 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440b0070 con 0x7f58540a4030 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: dump: 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:39.846 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: _finish_auth 0 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: _check_auth_tickets 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 5 monclient: authenticate success, global_id 22373 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 1 -- 172.21.15.138:0/699230522 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58440ab600 con 0x7f58540a4030 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient: get_monmap_and_config success 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 4 set_mon_vals no callback set 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 1 -- 172.21.15.138:0/699230522 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58440ab9d0 con 0x7f58540a4030 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:39.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5851d9b640 10 monclient: dump: 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient: shutdown 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- 172.21.15.138:0/699230522 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4030 msgr2=0x7f58540a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:39.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 --2- 172.21.15.138:0/699230522 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4030 0x7f58540a3ab0 secure :-1 s=READY pgs=5667 cs=0 l=1 rev1=1 crypto rx=0x7f5844009870 tx=0x7f584401c360 comp rx=0 tx=0).stop 2024-09-11T05:11:39.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- 172.21.15.138:0/699230522 shutdown_connections 2024-09-11T05:11:39.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 --2- 172.21.15.138:0/699230522 >> v2:172.21.15.138:3300/0 conn(0x7f585413da40 0x7f585413fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:39.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 --2- 172.21.15.138:0/699230522 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4030 0x7f58540a3ab0 unknown :-1 s=CLOSED pgs=5667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:39.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- 172.21.15.138:0/699230522 >> 172.21.15.138:0/699230522 conn(0x7f58540ab940 msgr2=0x7f58540a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:39.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- 172.21.15.138:0/699230522 shutdown_connections 2024-09-11T05:11:39.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- 172.21.15.138:0/699230522 wait complete. 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient: build_initial_monmap 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient: monmap: 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:39.854 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 librados: starting msgr at 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 librados: starting objecter 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 Processor -- start 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- start start 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 librados: setting wanted keys 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 librados: calling monclient init 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient: init 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58540a4030 0x7f5854125b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:39.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient(hunting): picked mon.noname-b con 0x7f58540a4030 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4410 0x7f585413c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient(hunting): picked mon.noname-c con 0x7f58540a4410 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 0x7f5854123540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient(hunting): picked mon.noname-a con 0x7f585413da40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5854143060 con 0x7f585413da40 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5854142ee0 con 0x7f58540a4030 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:39.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f58541431e0 con 0x7f58540a4410 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5858c7e640 10 monclient(hunting): authenticate will time out at 1183.283595s 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4410 0x7f585413c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 0x7f5854123540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4410 0x7f585413c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48412/0 (socket says 172.21.15.138:48412) 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 1 -- 172.21.15.138:0/233659006 learned_addr learned my addr 172.21.15.138:0/233659006 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 10 monclient(hunting): get_auth_request con 0x7f58540a4410 auth_method 0 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 10 monclient(hunting): get_auth_request con 0x7f585413da40 auth_method 0 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5849d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 1 -- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 msgr2=0x7f5854123540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 1 -- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 msgr2=0x7f5854123540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 0x7f5854123540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f5852d9d640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 0x7f5854123540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f585259c640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58540a4030 0x7f5854125b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f585259c640 10 monclient(hunting): get_auth_request con 0x7f58540a4030 auth_method 0 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f585259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f585259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.841+0000 7f585259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 10 monclient(hunting): handle_auth_done global_id 22376 payload 824 2024-09-11T05:11:39.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 1 -- 172.21.15.138:0/233659006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58540a4030 msgr2=0x7f5854125b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58540a4030 0x7f5854125b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 1 -- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 msgr2=0x7f5854123540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 0x7f5854123540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 1 monclient: found mon.noname-c 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 1 -- 172.21.15.138:0/233659006 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5854127360 con 0x7f58540a4410 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 1 -- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4410 msgr2=0x7f585413c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5849d9b640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4410 0x7f585413c9d0 secure :-1 s=READY pgs=5668 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c480 tx=0x7f583c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 client.?.objecter ms_handle_connect 0x7f58540a4410 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 1 -- 172.21.15.138:0/233659006 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c012a40 con 0x7f58540a4410 2024-09-11T05:11:39.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: dump: 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:11:39.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 5 monclient: authenticate success, global_id 22376 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 1 -- 172.21.15.138:0/233659006 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f583c017070 con 0x7f58540a4410 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 10 monclient: _renew_subs 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f585413d670 con 0x7f58540a4410 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 1 -- 172.21.15.138:0/233659006 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583c012e40 con 0x7f58540a4410 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 monclient: dump: 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:39.861 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5850d99640 4 set_mon_vals no callback set 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 10 client.22376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 10 monclient: _renew_subs 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58541d12a0 con 0x7f58540a4410 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 1 librados: init done 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:39.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58540a3ab0 con 0x7f58540a4410 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 1 -- 172.21.15.138:0/233659006 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f583c01c460 con 0x7f58540a4410 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 1 --2- 172.21.15.138:0/233659006 >> v2:172.21.15.190:6800/1562782189 conn(0x7f583403ced0 0x7f583403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 1 -- 172.21.15.138:0/233659006 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(181..181 src has 1..181) ==== 7044+0+0 (secure 0 0 0) 0x7f583c04ff10 con 0x7f58540a4410 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 10 client.22376.objecter ms_dispatch 0x7f5854000ba0 osd_map(181..181 src has 1..181) 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 3 client.22376.objecter handle_osd_map got epochs [181,181] > 0 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 3 client.22376.objecter handle_osd_map decoding full epoch 181 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.845+0000 7f584b7fe640 20 client.22376.objecter dump_active .. 0 homeless 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.849+0000 7f585259c640 1 --2- 172.21.15.138:0/233659006 >> v2:172.21.15.190:6800/1562782189 conn(0x7f583403ced0 0x7f583403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:39.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.849+0000 7f585259c640 10 monclient: get_auth_request con 0x7f583403ced0 auth_method 0 2024-09-11T05:11:39.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.849+0000 7f584b7fe640 10 client.22376.objecter ms_handle_connect 0x7f583403ced0 2024-09-11T05:11:39.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.849+0000 7f585259c640 1 --2- 172.21.15.138:0/233659006 >> v2:172.21.15.190:6800/1562782189 conn(0x7f583403ced0 0x7f583403f390 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7f58540708b0 tx=0x7f5844008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:39.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.853+0000 7f584b7fe640 1 -- 172.21.15.138:0/233659006 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f583c01ac60 con 0x7f58540a4410 2024-09-11T05:11:39.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.853+0000 7f584b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:39.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:39.853+0000 7f584b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:11:40.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.089+0000 7f5858c7e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:11:40.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.089+0000 7f5858c7e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:40.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.089+0000 7f5858c7e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:40.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.089+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5854124240 con 0x7f58540a4410 2024-09-11T05:11:40.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.089+0000 7f584b7fe640 1 -- 172.21.15.138:0/233659006 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f583c01c960 con 0x7f58540a4410 2024-09-11T05:11:40.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.089+0000 7f584b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:40.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.089+0000 7f584b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:11:40.096 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":95,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 10 librados: watch_flush enter 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 10 librados: watch_flush exit 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 20 client.22376.objecter shutdown clearing up homeless session... 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 10 client.22376.objecter successfully canceled tick 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 >> v2:172.21.15.190:6800/1562782189 conn(0x7f583403ced0 msgr2=0x7f583403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 --2- 172.21.15.138:0/233659006 >> v2:172.21.15.190:6800/1562782189 conn(0x7f583403ced0 0x7f583403f390 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7f58540708b0 tx=0x7f5844008040 comp rx=0 tx=0).stop 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 10 monclient: shutdown 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4410 msgr2=0x7f585413c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:40.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4410 0x7f585413c9d0 secure :-1 s=READY pgs=5668 cs=0 l=1 rev1=1 crypto rx=0x7f583c00c480 tx=0x7f583c00c940 comp rx=0 tx=0).stop 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 shutdown_connections 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 --2- 172.21.15.138:0/233659006 >> v2:172.21.15.190:6800/1562782189 conn(0x7f583403ced0 0x7f583403f390 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f585413da40 0x7f5854123540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f58540a4410 0x7f585413c9d0 unknown :-1 s=CLOSED pgs=5668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 --2- 172.21.15.138:0/233659006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f58540a4030 0x7f5854125b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 >> 172.21.15.138:0/233659006 conn(0x7f58540ab940 msgr2=0x7f585413eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 shutdown_connections 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 -- 172.21.15.138:0/233659006 wait complete. 2024-09-11T05:11:40.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:40.093+0000 7f5858c7e640 1 librados: shutdown 2024-09-11T05:11:43.116 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:11:43.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient: get_monmap_and_config 2024-09-11T05:11:43.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient: build_initial_monmap 2024-09-11T05:11:43.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient: monmap: 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:11:43.246893+0000 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:11:43.246893+0000 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:11:43.246 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:43.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 1 Processor -- start 2024-09-11T05:11:43.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 1 -- start start 2024-09-11T05:11:43.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient: init 2024-09-11T05:11:43.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:43.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-11T05:11:43.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 0x7f1d10136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient(hunting): picked mon.noname-a con 0x7f1d1006e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:11:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d10136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient(hunting): picked mon.noname-b con 0x7f1d10136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d100a1920 addr v1:172.21.15.138:6789/0 2024-09-11T05:11:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d100a3270 con 0x7f1d100a1920 2024-09-11T05:11:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f1d1006e4c0 con 0x7f1d1006e0e0 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1d100a0af0 con 0x7f1d10136170 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d174aa640 10 monclient(hunting): authenticate will time out at 1186.682935s 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d10136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 0x7f1d10136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d10136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48430/0 (socket says 172.21.15.138:48430) 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 1 -- 172.21.15.138:0/2823959999 learned_addr learned my addr 172.21.15.138:0/2823959999 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient(hunting): get_auth_request con 0x7f1d10136170 auth_method 0 2024-09-11T05:11:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 10 monclient(hunting): get_auth_request con 0x7f1d1006e0e0 auth_method 0 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15a47640 1 --1- 172.21.15.138:0/2823959999 >> v1:172.21.15.138:6789/0 conn(0x7f1d100a1920 0x7f1d100a1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38430/0 (socket says 172.21.15.138:38430) 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 1 -- 172.21.15.138:0/2823959999 >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 msgr2=0x7f1d10136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 1 -- 172.21.15.138:0/2823959999 >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 msgr2=0x7f1d10136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 1 --2- 172.21.15.138:0/2823959999 >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 0x7f1d10136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d15246640 1 --2- 172.21.15.138:0/2823959999 >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 0x7f1d10136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient(hunting): handle_auth_done global_id 22382 payload 274 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 1 -- 172.21.15.138:0/2823959999 >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 msgr2=0x7f1d10136720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 1 --2- 172.21.15.138:0/2823959999 >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 0x7f1d10136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 1 -- 172.21.15.138:0/2823959999 >> v1:172.21.15.138:6789/0 conn(0x7f1d100a1920 legacy=0x7f1d100a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient: _finish_hunting 0 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 1 monclient: found mon.noname-b 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:11:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.241+0000 7f1d14a45640 1 -- 172.21.15.138:0/2823959999 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d100a2f90 con 0x7f1d10136170 2024-09-11T05:11:43.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.245+0000 7f1d14a45640 1 -- 172.21.15.138:0/2823959999 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 msgr2=0x7f1d10136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:43.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.245+0000 7f1d14a45640 1 --2- 172.21.15.138:0/2823959999 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d10136c60 secure :-1 s=READY pgs=5672 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009870 tx=0x7f1d0c01c240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30292b337d028c16 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:43.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.245+0000 7f1d07fff640 1 -- 172.21.15.138:0/2823959999 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0b0070 con 0x7f1d10136170 2024-09-11T05:11:43.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.245+0000 7f1d07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 10 monclient: dump: 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 10 monclient: _finish_auth 0 2024-09-11T05:11:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 10 monclient: _check_auth_tickets 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 1 -- 172.21.15.138:0/2823959999 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d0c0ab600 con 0x7f1d10136170 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 5 monclient: authenticate success, global_id 22382 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient: get_monmap_and_config success 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 4 set_mon_vals no callback set 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 1 -- 172.21.15.138:0/2823959999 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0ab9d0 con 0x7f1d10136170 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d07fff640 10 monclient: dump: 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient: shutdown 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- 172.21.15.138:0/2823959999 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 msgr2=0x7f1d10136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 --2- 172.21.15.138:0/2823959999 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d10136c60 secure :-1 s=READY pgs=5672 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009870 tx=0x7f1d0c01c240 comp rx=0 tx=0).stop 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- 172.21.15.138:0/2823959999 shutdown_connections 2024-09-11T05:11:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 --2- 172.21.15.138:0/2823959999 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d10136c60 unknown :-1 s=CLOSED pgs=5672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 --2- 172.21.15.138:0/2823959999 >> v2:172.21.15.138:3300/0 conn(0x7f1d1006e0e0 0x7f1d10136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- 172.21.15.138:0/2823959999 >> 172.21.15.138:0/2823959999 conn(0x7f1d10063640 msgr2=0x7f1d10063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- 172.21.15.138:0/2823959999 shutdown_connections 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- 172.21.15.138:0/2823959999 wait complete. 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient: build_initial_monmap 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient: monmap: 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:11:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 librados: starting msgr at 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 librados: starting objecter 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 Processor -- start 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- start start 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 librados: setting wanted keys 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 librados: calling monclient init 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient: init 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 0x7f1d1012fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient(hunting): picked mon.noname-a con 0x7f1d1006e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d100a1920 0x7f1d10130020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient(hunting): picked mon.noname-b con 0x7f1d100a1920 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d101d07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient(hunting): picked mon.noname-c con 0x7f1d10136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1d101410a0 con 0x7f1d1006e0e0 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1d10140f20 con 0x7f1d100a1920 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1d10141220 con 0x7f1d10136170 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d174aa640 10 monclient(hunting): authenticate will time out at 1186.690091s 2024-09-11T05:11:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 0x7f1d1012fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 0x7f1d1012fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:41844/0 (socket says 172.21.15.138:41844) 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d101d07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 1 -- 172.21.15.138:0/1333680670 learned_addr learned my addr 172.21.15.138:0/1333680670 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 10 monclient(hunting): get_auth_request con 0x7f1d1006e0e0 auth_method 0 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient(hunting): get_auth_request con 0x7f1d10136170 auth_method 0 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 1 -- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 msgr2=0x7f1d1012fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 1 -- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 msgr2=0x7f1d1012fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:11:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 0x7f1d1012fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15246640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 0x7f1d1012fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d14a45640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d100a1920 0x7f1d10130020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient(hunting): handle_auth_done global_id 22385 payload 824 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 1 -- 172.21.15.138:0/1333680670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d100a1920 msgr2=0x7f1d10130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d100a1920 0x7f1d10130020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 1 -- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 msgr2=0x7f1d1012fae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 0x7f1d1012fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient: _finish_hunting 0 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 1 monclient: found mon.noname-c 2024-09-11T05:11:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:11:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d15a47640 1 -- 172.21.15.138:0/1333680670 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1013c5f0 con 0x7f1d10136170 2024-09-11T05:11:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d14a45640 10 monclient: get_auth_request con 0x7f1d100a1920 auth_method 0 2024-09-11T05:11:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.249+0000 7f1d14a45640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d100a1920 0x7f1d10130020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:11:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d15a47640 1 -- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 msgr2=0x7f1d101d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d15a47640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d101d07e0 secure :-1 s=READY pgs=5673 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800c780 tx=0x7f1d0800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 client.?.objecter ms_handle_connect 0x7f1d10136170 2024-09-11T05:11:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:11:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 1 -- 172.21.15.138:0/1333680670 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08017070 con 0x7f1d10136170 2024-09-11T05:11:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:11:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: dump: 2024-09-11T05:11:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: _finish_auth 0 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 1 -- 172.21.15.138:0/1333680670 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d0800a640 con 0x7f1d10136170 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 5 monclient: authenticate success, global_id 22385 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 10 monclient: _renew_subs 2024-09-11T05:11:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1009e110 con 0x7f1d10136170 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 1 -- 172.21.15.138:0/1333680670 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0800aa30 con 0x7f1d10136170 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 monclient: dump: 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d06ffd640 4 set_mon_vals no callback set 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 10 client.22385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 10 monclient: _renew_subs 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1013a290 con 0x7f1d10136170 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 1 librados: init done 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d100a1db0 con 0x7f1d10136170 2024-09-11T05:11:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 1 -- 172.21.15.138:0/1333680670 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1d0800ac10 con 0x7f1d10136170 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 1 --2- 172.21.15.138:0/1333680670 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1cf403cf00 0x7f1cf403f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 1 -- 172.21.15.138:0/1333680670 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(184..184 src has 1..184) ==== 5462+0+0 (secure 0 0 0) 0x7f1d0804f220 con 0x7f1d10136170 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 10 client.22385.objecter ms_dispatch 0x7f1d10000ba0 osd_map(184..184 src has 1..184) 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 3 client.22385.objecter handle_osd_map got epochs [184,184] > 0 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 3 client.22385.objecter handle_osd_map decoding full epoch 184 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d05ffb640 20 client.22385.objecter dump_active .. 0 homeless 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d15246640 1 --2- 172.21.15.138:0/1333680670 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1cf403cf00 0x7f1cf403f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.253+0000 7f1d15246640 10 monclient: get_auth_request con 0x7f1cf403cf00 auth_method 0 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.257+0000 7f1d05ffb640 10 client.22385.objecter ms_handle_connect 0x7f1cf403cf00 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.257+0000 7f1d15246640 1 --2- 172.21.15.138:0/1333680670 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1cf403cf00 0x7f1cf403f3c0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f1d00006e90 tx=0x7f1d000091a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.257+0000 7f1d05ffb640 1 -- 172.21.15.138:0/1333680670 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f1d080079b0 con 0x7f1d10136170 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.257+0000 7f1d05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.257+0000 7f1d05ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:11:43.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:11:43.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:43.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:43.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1d1006ccd0 con 0x7f1d10136170 2024-09-11T05:11:43.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d05ffb640 1 -- 172.21.15.138:0/1333680670 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f1d08019cd0 con 0x7f1d10136170 2024-09-11T05:11:43.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:43.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d05ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:11:43.505 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:11:43.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 10 librados: watch_flush enter 2024-09-11T05:11:43.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 10 librados: watch_flush exit 2024-09-11T05:11:43.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 20 client.22385.objecter shutdown clearing up homeless session... 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 10 client.22385.objecter successfully canceled tick 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1cf403cf00 msgr2=0x7f1cf403f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 --2- 172.21.15.138:0/1333680670 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1cf403cf00 0x7f1cf403f3c0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f1d00006e90 tx=0x7f1d000091a0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 10 monclient: shutdown 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 msgr2=0x7f1d101d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d101d07e0 secure :-1 s=READY pgs=5673 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800c780 tx=0x7f1d0800cc40 comp rx=0 tx=0).stop 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 shutdown_connections 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 --2- 172.21.15.138:0/1333680670 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1cf403cf00 0x7f1cf403f3c0 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1d10136170 0x7f1d101d07e0 unknown :-1 s=CLOSED pgs=5673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1d100a1920 0x7f1d10130020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 --2- 172.21.15.138:0/1333680670 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1d1006e0e0 0x7f1d1012fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:43.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 >> 172.21.15.138:0/1333680670 conn(0x7f1d10063640 msgr2=0x7f1d1013d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:43.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 shutdown_connections 2024-09-11T05:11:43.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 -- 172.21.15.138:0/1333680670 wait complete. 2024-09-11T05:11:43.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:43.501+0000 7f1d174aa640 1 librados: shutdown 2024-09-11T05:11:46.524 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:11:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.645+0000 7f8567da4640 10 monclient: get_monmap_and_config 2024-09-11T05:11:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.645+0000 7f8567da4640 10 monclient: build_initial_monmap 2024-09-11T05:11:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.645+0000 7f8567da4640 10 monclient: monmap: 2024-09-11T05:11:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:46.651 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:11:46.652695+0000 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:11:46.652695+0000 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:11:46.652 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 1 Processor -- start 2024-09-11T05:11:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 1 -- start start 2024-09-11T05:11:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient: init 2024-09-11T05:11:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-11T05:11:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f85600a5b40 0x7f85600a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient(hunting): picked mon.noname-a con 0x7f85600a5b40 addr v2:172.21.15.138:3300/0 2024-09-11T05:11:46.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85600a7130 addr v1:172.21.15.138:6789/0 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f856009d840 addr v1:172.21.15.190:6789/0 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f856009d490 con 0x7f85600a7130 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8560139720 con 0x7f856009d840 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f856009e110 con 0x7f85600a5b40 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8567da4640 10 monclient(hunting): authenticate will time out at 1190.089109s 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f85600a5b40 0x7f85600a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f856533f640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f85600a7130 0x7f8560138900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38444/0 (socket says 172.21.15.138:38444) 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f856533f640 1 -- 172.21.15.138:0/2869030310 learned_addr learned my addr 172.21.15.138:0/2869030310 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 10 monclient(hunting): get_auth_request con 0x7f85600a5b40 auth_method 0 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:46.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 1 -- 172.21.15.138:0/2869030310 >> v2:172.21.15.138:3300/0 conn(0x7f85600a5b40 msgr2=0x7f85600a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:11:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 1 -- 172.21.15.138:0/2869030310 >> v2:172.21.15.138:3300/0 conn(0x7f85600a5b40 msgr2=0x7f85600a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:11:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 1 --2- 172.21.15.138:0/2869030310 >> v2:172.21.15.138:3300/0 conn(0x7f85600a5b40 0x7f85600a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:46.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8565b40640 1 --2- 172.21.15.138:0/2869030310 >> v2:172.21.15.138:3300/0 conn(0x7f85600a5b40 0x7f85600a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f856533f640 1 -- 172.21.15.138:0/2869030310 >> v1:172.21.15.138:6789/0 conn(0x7f85600a7130 legacy=0x7f8560138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.649+0000 7f8566341640 1 -- 172.21.15.138:0/2869030310 >> v1:172.21.15.190:6789/0 conn(0x7f856009d840 legacy=0x7f8560139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1548271234 0 0) 0x7f8560139720 con 0x7f856009d840 2024-09-11T05:11:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): _negotiate 2024-09-11T05:11:46.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): my global_id is 20527 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f854c003530 con 0x7f856009d840 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 191385902 0 0) 0x7f854c003530 con 0x7f856009d840 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 >> v2:172.21.15.138:3300/0 conn(0x7f85600a5b40 msgr2=0x7f85600a5f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 --2- 172.21.15.138:0/2869030310 >> v2:172.21.15.138:3300/0 conn(0x7f85600a5b40 0x7f85600a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 >> v1:172.21.15.138:6789/0 conn(0x7f85600a7130 legacy=0x7f8560138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: _finish_hunting 0 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 monclient: found mon.noname-c-legacy 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.190:6789/0 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8560070580 con 0x7f856009d840 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: _finish_auth 0 2024-09-11T05:11:46.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: _check_auth_tickets 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f8558001d80 con 0x7f856009d840 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: dump: 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:46.660 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] but i'm connected to v1:172.21.15.190:6789/0, reconnecting 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 >> v1:172.21.15.190:6789/0 conn(0x7f856009d840 legacy=0x7f8560139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f854c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): picked mon.c con 0x7f854c003b30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f854c004810 0x7f854c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): picked mon.a con 0x7f854c004810 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f854c0099c0 0x7f854c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): picked mon.b con 0x7f854c0099c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:46.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f856009e110 con 0x7f854c004810 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f854c00c3d0 con 0x7f854c0099c0 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f854c00c5d0 con 0x7f854c003b30 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8564b3e640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f854c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f856533f640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f854c004810 0x7f854c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient(hunting): get_auth_request con 0x7f854c003b30 auth_method 0 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f856533f640 10 monclient(hunting): get_auth_request con 0x7f854c004810 auth_method 0 2024-09-11T05:11:46.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f856533f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f856533f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f856533f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f856533f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f856533f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f856533f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8566341640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f854c0099c0 0x7f854c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8566341640 10 monclient(hunting): get_auth_request con 0x7f854c0099c0 auth_method 0 2024-09-11T05:11:46.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8566341640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8566341640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8566341640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient(hunting): handle_auth_done global_id 20527 payload 293 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 1 -- 172.21.15.138:0/2869030310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f854c0099c0 msgr2=0x7f854c00bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f854c0099c0 0x7f854c00bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 1 -- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f854c004810 msgr2=0x7f854c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f854c004810 0x7f854c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient: _finish_hunting 0 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 1 monclient: found mon.c 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 1 -- 172.21.15.138:0/2869030310 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c003530 con 0x7f854c003b30 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient: _finish_auth 0 2024-09-11T05:11:46.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 10 monclient: _check_auth_tickets 2024-09-11T05:11:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8565b40640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8567da4640 5 monclient: authenticate success, global_id 20527 2024-09-11T05:11:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8567da4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:11:46.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.653+0000 7f8566341640 1 -- 172.21.15.138:0/2869030310 reap_dead start 2024-09-11T05:11:46.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8565b40640 1 -- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 msgr2=0x7f854c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8565b40640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f854c003f50 secure :-1 s=READY pgs=5677 cs=0 l=1 rev1=1 crypto rx=0x7f855c00c820 tx=0x7f855c01c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6069504fc28b909e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c00f040 con 0x7f854c003b30 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 10 monclient: dump: 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:46.672 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f855c0b0070 con 0x7f854c003b30 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 1 -- 172.21.15.138:0/2869030310 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c0ab670 con 0x7f854c003b30 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient: get_monmap_and_config success 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 4 set_mon_vals no callback set 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8564b3e640 10 monclient: dump: 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:46.673 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient: shutdown 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 msgr2=0x7f854c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f854c003f50 secure :-1 s=READY pgs=5677 cs=0 l=1 rev1=1 crypto rx=0x7f855c00c820 tx=0x7f855c01c4f0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- 172.21.15.138:0/2869030310 shutdown_connections 2024-09-11T05:11:46.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 --2- 172.21.15.138:0/2869030310 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f854c003f50 unknown :-1 s=CLOSED pgs=5677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- 172.21.15.138:0/2869030310 >> 172.21.15.138:0/2869030310 conn(0x7f85600aba10 msgr2=0x7f85600a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- 172.21.15.138:0/2869030310 shutdown_connections 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- 172.21.15.138:0/2869030310 wait complete. 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient: build_initial_monmap 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient: monmap: 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:11:46.675 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 librados: starting msgr at 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 librados: starting objecter 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 Processor -- start 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- start start 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 librados: setting wanted keys 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 librados: calling monclient init 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient: init 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f85601d0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient(hunting): picked mon.noname-c con 0x7f854c003b30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8560137bd0 0x7f8560138050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient(hunting): picked mon.noname-b con 0x7f8560137bd0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:11:46.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 0x7f8560121be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient(hunting): picked mon.noname-a con 0x7f8560121790 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8560138900 con 0x7f8560121790 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f85600a6940 con 0x7f8560137bd0 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8560138a80 con 0x7f854c003b30 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8567da4640 10 monclient(hunting): authenticate will time out at 1190.099913s 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.657+0000 7f8565b40640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f85601d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f85601d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48450/0 (socket says 172.21.15.138:48450) 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 -- 172.21.15.138:0/244546927 learned_addr learned my addr 172.21.15.138:0/244546927 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient(hunting): get_auth_request con 0x7f854c003b30 auth_method 0 2024-09-11T05:11:46.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 0x7f8560121be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 10 monclient(hunting): get_auth_request con 0x7f8560121790 auth_method 0 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 1 -- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 msgr2=0x7f8560121be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 1 -- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 msgr2=0x7f8560121be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 0x7f8560121be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:46.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8566341640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 0x7f8560121be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f856533f640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8560137bd0 0x7f8560138050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f856533f640 10 monclient(hunting): get_auth_request con 0x7f8560137bd0 auth_method 0 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f856533f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f856533f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f856533f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient(hunting): handle_auth_done global_id 22394 payload 824 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 -- 172.21.15.138:0/244546927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8560137bd0 msgr2=0x7f8560138050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8560137bd0 0x7f8560138050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 -- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 msgr2=0x7f8560121be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 0x7f8560121be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient: _finish_hunting 0 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 monclient: found mon.noname-c 2024-09-11T05:11:46.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 -- 172.21.15.138:0/244546927 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8560138590 con 0x7f854c003b30 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 -- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 msgr2=0x7f85601d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8565b40640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f85601d0300 secure :-1 s=READY pgs=5678 cs=0 l=1 rev1=1 crypto rx=0x7f855c0ad040 tx=0x7f854c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 client.?.objecter ms_handle_connect 0x7f854c003b30 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 1 -- 172.21.15.138:0/244546927 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c0abbe0 con 0x7f854c003b30 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: dump: 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.680 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: _finish_auth 0 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: _check_auth_tickets 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 1 -- 172.21.15.138:0/244546927 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f855c0abd80 con 0x7f854c003b30 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 5 monclient: authenticate success, global_id 22394 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 10 monclient: _renew_subs 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:46.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8560124cf0 con 0x7f854c003b30 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 1 -- 172.21.15.138:0/244546927 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c0c06c0 con 0x7f854c003b30 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 10 monclient: dump: 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:46.682 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8553fff640 4 set_mon_vals no callback set 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 10 client.22394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 10 monclient: _renew_subs 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8560125590 con 0x7f854c003b30 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 1 librados: init done 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f85527fc640 1 -- 172.21.15.138:0/244546927 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f855c0c8030 con 0x7f854c003b30 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.661+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8560138d80 con 0x7f854c003b30 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f85527fc640 1 --2- 172.21.15.138:0/244546927 >> v2:172.21.15.190:6800/1562782189 conn(0x7f854403cf00 0x7f854403f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:46.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f85527fc640 1 -- 172.21.15.138:0/244546927 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(188..188 src has 1..188) ==== 7044+0+0 (secure 0 0 0) 0x7f855c0ea630 con 0x7f854c003b30 2024-09-11T05:11:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f85527fc640 10 client.22394.objecter ms_dispatch 0x7f8560000ba0 osd_map(188..188 src has 1..188) 2024-09-11T05:11:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f85527fc640 3 client.22394.objecter handle_osd_map got epochs [188,188] > 0 2024-09-11T05:11:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f85527fc640 3 client.22394.objecter handle_osd_map decoding full epoch 188 2024-09-11T05:11:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f85527fc640 20 client.22394.objecter dump_active .. 0 homeless 2024-09-11T05:11:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f856533f640 1 --2- 172.21.15.138:0/244546927 >> v2:172.21.15.190:6800/1562782189 conn(0x7f854403cf00 0x7f854403f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f856533f640 10 monclient: get_auth_request con 0x7f854403cf00 auth_method 0 2024-09-11T05:11:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f85527fc640 10 client.22394.objecter ms_handle_connect 0x7f854403cf00 2024-09-11T05:11:46.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.665+0000 7f856533f640 1 --2- 172.21.15.138:0/244546927 >> v2:172.21.15.190:6800/1562782189 conn(0x7f854403cf00 0x7f854403f3c0 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f8554004e90 tx=0x7f8554009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:46.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.669+0000 7f85527fc640 1 -- 172.21.15.138:0/244546927 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f8560124cf0 con 0x7f854c003b30 2024-09-11T05:11:46.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.669+0000 7f85527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:46.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.669+0000 7f85527fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:11:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:11:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:46.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f85601d0840 con 0x7f854c003b30 2024-09-11T05:11:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f85527fc640 1 -- 172.21.15.138:0/244546927 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f855c016200 con 0x7f854c003b30 2024-09-11T05:11:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f85527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:46.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f85527fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:11:46.909 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":102,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:11:46.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 10 librados: watch_flush enter 2024-09-11T05:11:46.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 10 librados: watch_flush exit 2024-09-11T05:11:46.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 20 client.22394.objecter shutdown clearing up homeless session... 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 10 client.22394.objecter successfully canceled tick 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 >> v2:172.21.15.190:6800/1562782189 conn(0x7f854403cf00 msgr2=0x7f854403f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 --2- 172.21.15.138:0/244546927 >> v2:172.21.15.190:6800/1562782189 conn(0x7f854403cf00 0x7f854403f3c0 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f8554004e90 tx=0x7f8554009040 comp rx=0 tx=0).stop 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 10 monclient: shutdown 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 msgr2=0x7f85601d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f85601d0300 secure :-1 s=READY pgs=5678 cs=0 l=1 rev1=1 crypto rx=0x7f855c0ad040 tx=0x7f854c005840 comp rx=0 tx=0).stop 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 shutdown_connections 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 --2- 172.21.15.138:0/244546927 >> v2:172.21.15.190:6800/1562782189 conn(0x7f854403cf00 0x7f854403f3c0 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8560121790 0x7f8560121be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8560137bd0 0x7f8560138050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 --2- 172.21.15.138:0/244546927 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f854c003b30 0x7f85601d0300 unknown :-1 s=CLOSED pgs=5678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:46.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 >> 172.21.15.138:0/244546927 conn(0x7f85600aba10 msgr2=0x7f85600a4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:46.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 shutdown_connections 2024-09-11T05:11:46.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 -- 172.21.15.138:0/244546927 wait complete. 2024-09-11T05:11:46.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:46.905+0000 7f8567da4640 1 librados: shutdown 2024-09-11T05:11:49.928 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:11:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient: get_monmap_and_config 2024-09-11T05:11:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient: build_initial_monmap 2024-09-11T05:11:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient: monmap: 2024-09-11T05:11:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:50.057 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:11:50.058608+0000 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:11:50.058608+0000 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:11:50.058 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 1 Processor -- start 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 1 -- start start 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient: init 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f783006e0e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7830136e30 addr v1:172.21.15.138:6789/0 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136170 0x7f783013ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient(hunting): picked mon.noname-b con 0x7f7830136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:50.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78300a3270 con 0x7f7830136e30 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78300a0af0 con 0x7f783006e0e0 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f783006e4c0 con 0x7f7830136170 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f783803b640 10 monclient(hunting): authenticate will time out at 1193.494810s 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78355d6640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f7830136e30 0x7f783013b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:47370/0 (socket says 172.21.15.138:47370) 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136170 0x7f783013ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78355d6640 1 -- 172.21.15.138:0/2597313722 learned_addr learned my addr 172.21.15.138:0/2597313722 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient(hunting): get_auth_request con 0x7f7830136170 auth_method 0 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:50.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient(hunting): handle_auth_done global_id 22403 payload 274 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 1 -- 172.21.15.138:0/2597313722 >> v1:172.21.15.190:6789/0 conn(0x7f783006e0e0 legacy=0x7f7830136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 1 -- 172.21.15.138:0/2597313722 >> v1:172.21.15.138:6789/0 conn(0x7f7830136e30 legacy=0x7f783013b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient: _finish_hunting 0 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 1 monclient: found mon.noname-b 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:11:50.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.053+0000 7f78365d8640 1 -- 172.21.15.138:0/2597313722 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78300a2f90 con 0x7f7830136170 2024-09-11T05:11:50.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f78365d8640 1 -- 172.21.15.138:0/2597313722 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136170 msgr2=0x7f783013ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:50.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f78365d8640 1 --2- 172.21.15.138:0/2597313722 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136170 0x7f783013ec00 secure :-1 s=READY pgs=5681 cs=0 l=1 rev1=1 crypto rx=0x7f78280097f0 tx=0x7f782801c250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff94981aeacd5f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:50.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 1 -- 172.21.15.138:0/2597313722 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280b0070 con 0x7f7830136170 2024-09-11T05:11:50.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:50.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:11:50.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: dump: 2024-09-11T05:11:50.064 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: _finish_auth 0 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: _check_auth_tickets 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f783803b640 5 monclient: authenticate success, global_id 22403 2024-09-11T05:11:50.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 1 -- 172.21.15.138:0/2597313722 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78280ab600 con 0x7f7830136170 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f783803b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f783803b640 10 monclient: get_monmap_and_config success 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f783803b640 4 set_mon_vals no callback set 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 1 -- 172.21.15.138:0/2597313722 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280ab9d0 con 0x7f7830136170 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.057+0000 7f7834dd5640 10 monclient: dump: 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:50.066 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient: shutdown 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- 172.21.15.138:0/2597313722 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136170 msgr2=0x7f783013ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 --2- 172.21.15.138:0/2597313722 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136170 0x7f783013ec00 secure :-1 s=READY pgs=5681 cs=0 l=1 rev1=1 crypto rx=0x7f78280097f0 tx=0x7f782801c250 comp rx=0 tx=0).stop 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- 172.21.15.138:0/2597313722 shutdown_connections 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 --2- 172.21.15.138:0/2597313722 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136170 0x7f783013ec00 unknown :-1 s=CLOSED pgs=5681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- 172.21.15.138:0/2597313722 >> 172.21.15.138:0/2597313722 conn(0x7f7830063640 msgr2=0x7f7830063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- 172.21.15.138:0/2597313722 shutdown_connections 2024-09-11T05:11:50.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- 172.21.15.138:0/2597313722 wait complete. 2024-09-11T05:11:50.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient: build_initial_monmap 2024-09-11T05:11:50.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient: monmap: 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 librados: starting msgr at 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 librados: starting objecter 2024-09-11T05:11:50.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 Processor -- start 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- start start 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 librados: setting wanted keys 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 librados: calling monclient init 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient: init 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f783006e0e0 0x7f78300734e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient(hunting): picked mon.noname-a con 0x7f783006e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7830136170 0x7f783013d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient(hunting): picked mon.noname-b con 0x7f7830136170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136e30 0x7f783013d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient(hunting): picked mon.noname-c con 0x7f7830136e30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:50.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7830141d80 con 0x7f783006e0e0 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7830141c00 con 0x7f7830136170 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7830141f00 con 0x7f7830136e30 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f783803b640 10 monclient(hunting): authenticate will time out at 1193.501933s 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136e30 0x7f783013d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136e30 0x7f783013d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:41278/0 (socket says 172.21.15.138:41278) 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 -- 172.21.15.138:0/1473446731 learned_addr learned my addr 172.21.15.138:0/1473446731 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient(hunting): get_auth_request con 0x7f7830136e30 auth_method 0 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:50.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f7835dd7640 1 --2- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f783006e0e0 0x7f78300734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f7835dd7640 10 monclient(hunting): get_auth_request con 0x7f783006e0e0 auth_method 0 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f7835dd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f7835dd7640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f7835dd7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient(hunting): handle_auth_done global_id 22406 payload 824 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 -- 172.21.15.138:0/1473446731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7830136170 msgr2=0x7f783013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 --2- 172.21.15.138:0/1473446731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7830136170 0x7f783013d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 -- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f783006e0e0 msgr2=0x7f78300734e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 --2- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f783006e0e0 0x7f78300734e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient: _finish_hunting 0 2024-09-11T05:11:50.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 monclient: found mon.noname-c 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.061+0000 7f78365d8640 1 -- 172.21.15.138:0/1473446731 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78301cc680 con 0x7f7830136e30 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78365d8640 1 -- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136e30 msgr2=0x7f783013d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78365d8640 1 --2- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136e30 0x7f783013d930 secure :-1 s=READY pgs=5682 cs=0 l=1 rev1=1 crypto rx=0x7f78280097f0 tx=0x7f782801aa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 client.?.objecter ms_handle_connect 0x7f7830136e30 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 1 -- 172.21.15.138:0/1473446731 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280b0050 con 0x7f7830136e30 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: dump: 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:50.075 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: _finish_auth 0 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: _check_auth_tickets 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 5 monclient: authenticate success, global_id 22406 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 10 monclient: _renew_subs 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 1 -- 172.21.15.138:0/1473446731 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78280bc070 con 0x7f7830136e30 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:50.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7830139ec0 con 0x7f7830136e30 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 1 -- 172.21.15.138:0/1473446731 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280b5c60 con 0x7f7830136e30 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 monclient: dump: 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:50.077 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f7823fff640 4 set_mon_vals no callback set 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 10 client.22406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 10 monclient: _renew_subs 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7830073f10 con 0x7f7830136e30 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 1 librados: init done 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7830136720 con 0x7f7830136e30 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 1 -- 172.21.15.138:0/1473446731 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f78280c1460 con 0x7f7830136e30 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 1 --2- 172.21.15.138:0/1473446731 >> v2:172.21.15.190:6800/1562782189 conn(0x7f781403cef0 0x7f781403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 1 -- 172.21.15.138:0/1473446731 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(191..191 src has 1..191) ==== 5462+0+0 (secure 0 0 0) 0x7f78280ea5d0 con 0x7f7830136e30 2024-09-11T05:11:50.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 10 client.22406.objecter ms_dispatch 0x7f7830000ba0 osd_map(191..191 src has 1..191) 2024-09-11T05:11:50.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 3 client.22406.objecter handle_osd_map got epochs [191,191] > 0 2024-09-11T05:11:50.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 3 client.22406.objecter handle_osd_map decoding full epoch 191 2024-09-11T05:11:50.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f78227fc640 20 client.22406.objecter dump_active .. 0 homeless 2024-09-11T05:11:50.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f7835dd7640 1 --2- 172.21.15.138:0/1473446731 >> v2:172.21.15.190:6800/1562782189 conn(0x7f781403cef0 0x7f781403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:50.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.065+0000 7f7835dd7640 10 monclient: get_auth_request con 0x7f781403cef0 auth_method 0 2024-09-11T05:11:50.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.069+0000 7f78227fc640 10 client.22406.objecter ms_handle_connect 0x7f781403cef0 2024-09-11T05:11:50.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.069+0000 7f7835dd7640 1 --2- 172.21.15.138:0/1473446731 >> v2:172.21.15.190:6800/1562782189 conn(0x7f781403cef0 0x7f781403f3b0 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f782400b810 tx=0x7f7824008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:50.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.069+0000 7f78227fc640 1 -- 172.21.15.138:0/1473446731 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f78301cc680 con 0x7f7830136e30 2024-09-11T05:11:50.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.069+0000 7f78227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:50.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.069+0000 7f78227fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:11:50.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.305+0000 7f783803b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:11:50.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.305+0000 7f783803b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:50.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.305+0000 7f783803b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:50.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.305+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f783013de70 con 0x7f7830136e30 2024-09-11T05:11:50.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.305+0000 7f78227fc640 1 -- 172.21.15.138:0/1473446731 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f78300a3270 con 0x7f7830136e30 2024-09-11T05:11:50.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.305+0000 7f78227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:50.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.305+0000 7f78227fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:11:50.310 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":105,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:11:50.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.305+0000 7f783803b640 10 librados: watch_flush enter 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 10 librados: watch_flush exit 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 20 client.22406.objecter shutdown clearing up homeless session... 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 10 client.22406.objecter successfully canceled tick 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 >> v2:172.21.15.190:6800/1562782189 conn(0x7f781403cef0 msgr2=0x7f781403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 --2- 172.21.15.138:0/1473446731 >> v2:172.21.15.190:6800/1562782189 conn(0x7f781403cef0 0x7f781403f3b0 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7f782400b810 tx=0x7f7824008040 comp rx=0 tx=0).stop 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 10 monclient: shutdown 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136e30 msgr2=0x7f783013d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 --2- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136e30 0x7f783013d930 secure :-1 s=READY pgs=5682 cs=0 l=1 rev1=1 crypto rx=0x7f78280097f0 tx=0x7f782801aa90 comp rx=0 tx=0).stop 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 shutdown_connections 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 --2- 172.21.15.138:0/1473446731 >> v2:172.21.15.190:6800/1562782189 conn(0x7f781403cef0 0x7f781403f3b0 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 --2- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7830136e30 0x7f783013d930 unknown :-1 s=CLOSED pgs=5682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:50.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 --2- 172.21.15.138:0/1473446731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7830136170 0x7f783013d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:50.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 --2- 172.21.15.138:0/1473446731 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f783006e0e0 0x7f78300734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:50.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 >> 172.21.15.138:0/1473446731 conn(0x7f7830063640 msgr2=0x7f7830132b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:50.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 shutdown_connections 2024-09-11T05:11:50.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 -- 172.21.15.138:0/1473446731 wait complete. 2024-09-11T05:11:50.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:50.309+0000 7f783803b640 1 librados: shutdown 2024-09-11T05:11:53.330 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:11:53.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient: get_monmap_and_config 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient: build_initial_monmap 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient: monmap: 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:11:53.467031+0000 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:11:53.467031+0000 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:11:53.466 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:11:53.467 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:53.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 1 Processor -- start 2024-09-11T05:11:53.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 1 -- start start 2024-09-11T05:11:53.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient: init 2024-09-11T05:11:53.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:53.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T05:11:53.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdb640a5b40 0x7fdb640a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:53.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient(hunting): picked mon.noname-c con 0x7fdb640a5b40 addr v2:172.21.15.190:3300/0 2024-09-11T05:11:53.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb640a7130 0x7fdb640a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient(hunting): picked mon.noname-b con 0x7fdb640a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb6409d840 addr v1:172.21.15.190:6789/0 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb6409d490 con 0x7fdb6409d840 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdb6409e110 con 0x7fdb640a5b40 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdb640a69a0 con 0x7fdb640a7130 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb698db640 10 monclient(hunting): authenticate will time out at 1196.903058s 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb640a7130 0x7fdb640a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb640a7130 0x7fdb640a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:41292/0 (socket says 172.21.15.138:41292) 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 1 -- 172.21.15.138:0/624685854 learned_addr learned my addr 172.21.15.138:0/624685854 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:53.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient(hunting): get_auth_request con 0x7fdb640a7130 auth_method 0 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb62ffd640 1 --2- 172.21.15.138:0/624685854 >> v2:172.21.15.190:3300/0 conn(0x7fdb640a5b40 0x7fdb640a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb62ffd640 10 monclient(hunting): get_auth_request con 0x7fdb640a5b40 auth_method 0 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient(hunting): handle_auth_done global_id 22412 payload 274 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 1 -- 172.21.15.138:0/624685854 >> v2:172.21.15.190:3300/0 conn(0x7fdb640a5b40 msgr2=0x7fdb640a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 1 --2- 172.21.15.138:0/624685854 >> v2:172.21.15.190:3300/0 conn(0x7fdb640a5b40 0x7fdb640a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 1 -- 172.21.15.138:0/624685854 >> v1:172.21.15.190:6789/0 conn(0x7fdb6409d840 legacy=0x7fdb64138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:53.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient: _finish_hunting 0 2024-09-11T05:11:53.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 1 monclient: found mon.noname-b 2024-09-11T05:11:53.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:53.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:11:53.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.465+0000 7fdb627fc640 1 -- 172.21.15.138:0/624685854 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb64070580 con 0x7fdb640a7130 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb627fc640 1 -- 172.21.15.138:0/624685854 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb640a7130 msgr2=0x7fdb640a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb627fc640 1 --2- 172.21.15.138:0/624685854 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb640a7130 0x7fdb640a6460 secure :-1 s=READY pgs=5683 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009870 tx=0x7fdb4c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c022c120d2e2ca77 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 1 -- 172.21.15.138:0/624685854 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0b0070 con 0x7fdb640a7130 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: dump: 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:53.476 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: _finish_auth 0 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 1 -- 172.21.15.138:0/624685854 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4c0ab600 con 0x7fdb640a7130 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 5 monclient: authenticate success, global_id 22412 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:11:53.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 1 -- 172.21.15.138:0/624685854 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0a4e60 con 0x7fdb640a7130 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb61ffb640 10 monclient: dump: 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:53.478 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient: get_monmap_and_config success 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 4 set_mon_vals no callback set 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient: shutdown 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- 172.21.15.138:0/624685854 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb640a7130 msgr2=0x7fdb640a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 --2- 172.21.15.138:0/624685854 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb640a7130 0x7fdb640a6460 secure :-1 s=READY pgs=5683 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009870 tx=0x7fdb4c01c420 comp rx=0 tx=0).stop 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- 172.21.15.138:0/624685854 shutdown_connections 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 --2- 172.21.15.138:0/624685854 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb640a7130 0x7fdb640a6460 unknown :-1 s=CLOSED pgs=5683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 --2- 172.21.15.138:0/624685854 >> v2:172.21.15.190:3300/0 conn(0x7fdb640a5b40 0x7fdb640a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- 172.21.15.138:0/624685854 >> 172.21.15.138:0/624685854 conn(0x7fdb640aba10 msgr2=0x7fdb640a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- 172.21.15.138:0/624685854 shutdown_connections 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- 172.21.15.138:0/624685854 wait complete. 2024-09-11T05:11:53.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient: build_initial_monmap 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient: monmap: 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 librados: starting msgr at 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 librados: starting objecter 2024-09-11T05:11:53.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 Processor -- start 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- start start 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 librados: setting wanted keys 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 librados: calling monclient init 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient: init 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb6409d840 0x7fdb641d0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient(hunting): picked mon.noname-c con 0x7fdb6409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb640a5b40 0x7fdb64137bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient(hunting): picked mon.noname-b con 0x7fdb640a5b40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 0x7fdb6413cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient(hunting): picked mon.noname-a con 0x7fdb640a7130 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:11:53.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64142f60 con 0x7fdb640a7130 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64142de0 con 0x7fdb640a5b40 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdb641430e0 con 0x7fdb6409d840 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb698db640 10 monclient(hunting): authenticate will time out at 1196.910714s 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb6409d840 0x7fdb641d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 0x7fdb6413cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb6409d840 0x7fdb641d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:41304/0 (socket says 172.21.15.138:41304) 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 -- 172.21.15.138:0/488123798 learned_addr learned my addr 172.21.15.138:0/488123798 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient(hunting): get_auth_request con 0x7fdb6409d840 auth_method 0 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:53.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 10 monclient(hunting): get_auth_request con 0x7fdb640a7130 auth_method 0 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 1 -- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 msgr2=0x7fdb6413cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 1 -- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 msgr2=0x7fdb6413cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 0x7fdb6413cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb637fe640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 0x7fdb6413cb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb627fc640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb640a5b40 0x7fdb64137bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:53.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb627fc640 10 monclient(hunting): get_auth_request con 0x7fdb640a5b40 auth_method 0 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient(hunting): handle_auth_done global_id 22415 payload 824 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 -- 172.21.15.138:0/488123798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb640a5b40 msgr2=0x7fdb64137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb640a5b40 0x7fdb64137bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 -- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 msgr2=0x7fdb6413cb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 0x7fdb6413cb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 monclient: found mon.noname-c 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:11:53.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.473+0000 7fdb62ffd640 1 -- 172.21.15.138:0/488123798 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb640a4d40 con 0x7fdb6409d840 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb62ffd640 1 -- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb6409d840 msgr2=0x7fdb641d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb62ffd640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb6409d840 0x7fdb641d0530 secure :-1 s=READY pgs=5684 cs=0 l=1 rev1=1 crypto rx=0x7fdb54007ef0 tx=0x7fdb5400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 client.?.objecter ms_handle_connect 0x7fdb6409d840 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 1 -- 172.21.15.138:0/488123798 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54013790 con 0x7fdb6409d840 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: dump: 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:53.487 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: _finish_auth 0 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 1 -- 172.21.15.138:0/488123798 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb54019070 con 0x7fdb6409d840 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 1 -- 172.21.15.138:0/488123798 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54013b70 con 0x7fdb6409d840 2024-09-11T05:11:53.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 5 monclient: authenticate success, global_id 22415 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 10 monclient: _renew_subs 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6413e5a0 con 0x7fdb6409d840 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb617fa640 4 set_mon_vals no callback set 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 monclient: dump: 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:53.489 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 10 client.22415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 10 monclient: _renew_subs 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb64121fa0 con 0x7fdb6409d840 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 1 librados: init done 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 1 -- 172.21.15.138:0/488123798 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdb5401e460 con 0x7fdb6409d840 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:53.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb64138900 con 0x7fdb6409d840 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 1 --2- 172.21.15.138:0/488123798 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4403cea0 0x7fdb4403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 1 -- 172.21.15.138:0/488123798 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(195..195 src has 1..195) ==== 7036+0+0 (secure 0 0 0) 0x7fdb54050a60 con 0x7fdb6409d840 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 10 client.22415.objecter ms_dispatch 0x7fdb64000ba0 osd_map(195..195 src has 1..195) 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 3 client.22415.objecter handle_osd_map got epochs [195,195] > 0 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 3 client.22415.objecter handle_osd_map decoding full epoch 195 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb5bfff640 20 client.22415.objecter dump_active .. 0 homeless 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb627fc640 1 --2- 172.21.15.138:0/488123798 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4403cea0 0x7fdb4403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.477+0000 7fdb627fc640 10 monclient: get_auth_request con 0x7fdb4403cea0 auth_method 0 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.481+0000 7fdb5bfff640 10 client.22415.objecter ms_handle_connect 0x7fdb4403cea0 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.481+0000 7fdb627fc640 1 --2- 172.21.15.138:0/488123798 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4403cea0 0x7fdb4403f360 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7fdb6406d5a0 tx=0x7fdb4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.481+0000 7fdb5bfff640 1 -- 172.21.15.138:0/488123798 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fdb54052e20 con 0x7fdb6409d840 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.481+0000 7fdb5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:53.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.481+0000 7fdb5bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:11:53.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.721+0000 7fdb698db640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:11:53.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.721+0000 7fdb698db640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:53.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.721+0000 7fdb698db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:53.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.721+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdb640a2390 con 0x7fdb6409d840 2024-09-11T05:11:53.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.721+0000 7fdb5bfff640 1 -- 172.21.15.138:0/488123798 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fdb5404ee20 con 0x7fdb6409d840 2024-09-11T05:11:53.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.721+0000 7fdb5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:53.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.721+0000 7fdb5bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:11:53.723 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":109,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:11:53.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 10 librados: watch_flush enter 2024-09-11T05:11:53.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 10 librados: watch_flush exit 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 20 client.22415.objecter shutdown clearing up homeless session... 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 10 client.22415.objecter successfully canceled tick 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4403cea0 msgr2=0x7fdb4403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 --2- 172.21.15.138:0/488123798 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4403cea0 0x7fdb4403f360 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7fdb6406d5a0 tx=0x7fdb4c008040 comp rx=0 tx=0).stop 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 10 monclient: shutdown 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb6409d840 msgr2=0x7fdb641d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb6409d840 0x7fdb641d0530 secure :-1 s=READY pgs=5684 cs=0 l=1 rev1=1 crypto rx=0x7fdb54007ef0 tx=0x7fdb5400e4b0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 shutdown_connections 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 --2- 172.21.15.138:0/488123798 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdb4403cea0 0x7fdb4403f360 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdb640a7130 0x7fdb6413cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb640a5b40 0x7fdb64137bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 --2- 172.21.15.138:0/488123798 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdb6409d840 0x7fdb641d0530 unknown :-1 s=CLOSED pgs=5684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:53.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 >> 172.21.15.138:0/488123798 conn(0x7fdb640aba10 msgr2=0x7fdb640a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:53.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 shutdown_connections 2024-09-11T05:11:53.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 -- 172.21.15.138:0/488123798 wait complete. 2024-09-11T05:11:53.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:53.725+0000 7fdb698db640 1 librados: shutdown 2024-09-11T05:11:56.742 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:11:56.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 10 monclient: get_monmap_and_config 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 10 monclient: build_initial_monmap 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 10 monclient: monmap: 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:11:56.871944+0000 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:11:56.871944+0000 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:11:56.873 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:11:56.874 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:11:56.874 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:56.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 1 Processor -- start 2024-09-11T05:11:56.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 1 -- start start 2024-09-11T05:11:56.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 10 monclient: init 2024-09-11T05:11:56.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:56.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-11T05:11:56.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.869+0000 7f393a148640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 0x7f393409e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:56.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 10 monclient(hunting): picked mon.noname-a con 0x7f393406e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f393409e960 0x7f393409ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 10 monclient(hunting): picked mon.noname-b con 0x7f393409e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39341356f0 addr v1:172.21.15.190:6789/0 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f393406e4c0 con 0x7f39341356f0 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f393409f820 con 0x7f393406e0e0 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f393409f540 con 0x7f393409e960 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f393a148640 10 monclient(hunting): authenticate will time out at 1200.308084s 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f393409e960 0x7f393409ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 0x7f393409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:56.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f393409e960 0x7f393409ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:41306/0 (socket says 172.21.15.138:41306) 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 1 -- 172.21.15.138:0/1470828085 learned_addr learned my addr 172.21.15.138:0/1470828085 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 1 --2- 172.21.15.138:0/1470828085 >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 0x7f393409e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:58958/0 (socket says 172.21.15.138:58958) 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient(hunting): get_auth_request con 0x7f393409e960 auth_method 0 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 10 monclient(hunting): get_auth_request con 0x7f393406e0e0 auth_method 0 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:56.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 1 -- 172.21.15.138:0/1470828085 >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 msgr2=0x7f393409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 1 -- 172.21.15.138:0/1470828085 >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 msgr2=0x7f393409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 1 --2- 172.21.15.138:0/1470828085 >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 0x7f393409e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f39337fe640 1 --2- 172.21.15.138:0/1470828085 >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 0x7f393409e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient(hunting): handle_auth_done global_id 22421 payload 274 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 1 -- 172.21.15.138:0/1470828085 >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 msgr2=0x7f393409e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 1 --2- 172.21.15.138:0/1470828085 >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 0x7f393409e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 1 -- 172.21.15.138:0/1470828085 >> v1:172.21.15.190:6789/0 conn(0x7f39341356f0 legacy=0x7f3934135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 1 monclient: found mon.noname-b 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:11:56.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.873+0000 7f3932ffd640 1 -- 172.21.15.138:0/1470828085 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393409fb00 con 0x7f393409e960 2024-09-11T05:11:56.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3932ffd640 1 -- 172.21.15.138:0/1470828085 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f393409e960 msgr2=0x7f393409ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:56.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3932ffd640 1 --2- 172.21.15.138:0/1470828085 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f393409e960 0x7f393409ed40 secure :-1 s=READY pgs=5685 cs=0 l=1 rev1=1 crypto rx=0x7f3928009870 tx=0x7f392801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74c95ee4cdc69ef server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:56.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 1 -- 172.21.15.138:0/1470828085 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39280b0070 con 0x7f393409e960 2024-09-11T05:11:56.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: dump: 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:56.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: _finish_auth 0 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: _check_auth_tickets 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 1 -- 172.21.15.138:0/1470828085 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39280a4e50 con 0x7f393409e960 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 1 -- 172.21.15.138:0/1470828085 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39280ab820 con 0x7f393409e960 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 5 monclient: authenticate success, global_id 22421 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient: get_monmap_and_config success 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 4 set_mon_vals no callback set 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39327fc640 10 monclient: dump: 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:56.888 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient: shutdown 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- 172.21.15.138:0/1470828085 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f393409e960 msgr2=0x7f393409ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 --2- 172.21.15.138:0/1470828085 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f393409e960 0x7f393409ed40 secure :-1 s=READY pgs=5685 cs=0 l=1 rev1=1 crypto rx=0x7f3928009870 tx=0x7f392801c4b0 comp rx=0 tx=0).stop 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- 172.21.15.138:0/1470828085 shutdown_connections 2024-09-11T05:11:56.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 --2- 172.21.15.138:0/1470828085 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f393409e960 0x7f393409ed40 unknown :-1 s=CLOSED pgs=5685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 --2- 172.21.15.138:0/1470828085 >> v2:172.21.15.138:3300/0 conn(0x7f393406e0e0 0x7f393409e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- 172.21.15.138:0/1470828085 >> 172.21.15.138:0/1470828085 conn(0x7f3934063640 msgr2=0x7f3934063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- 172.21.15.138:0/1470828085 shutdown_connections 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- 172.21.15.138:0/1470828085 wait complete. 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient: build_initial_monmap 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient: monmap: 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:11:56.890 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 librados: starting msgr at 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 librados: starting objecter 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 Processor -- start 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- start start 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 librados: setting wanted keys 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 librados: calling monclient init 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient: init 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient: _reopen_session rank -1 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 0x7f3934122020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient(hunting): picked mon.noname-a con 0x7f393406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:11:56.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f393409e960 0x7f393411fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient(hunting): picked mon.noname-b con 0x7f393409e960 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39341356f0 0x7f393411ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient(hunting): picked mon.noname-c con 0x7f39341356f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f39340a2b40 con 0x7f393406e0e0 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f39340a29c0 con 0x7f393409e960 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient(hunting): start opening mon connection 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f39340a2cc0 con 0x7f39341356f0 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient(hunting): _renew_subs 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f393a148640 10 monclient(hunting): authenticate will time out at 1200.315062s 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 0x7f3934122020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39341356f0 0x7f393411ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 0x7f3934122020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:58968/0 (socket says 172.21.15.138:58968) 2024-09-11T05:11:56.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 1 -- 172.21.15.138:0/1558748622 learned_addr learned my addr 172.21.15.138:0/1558748622 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 10 monclient(hunting): get_auth_request con 0x7f393406e0e0 auth_method 0 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient(hunting): get_auth_request con 0x7f39341356f0 auth_method 0 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 1 -- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 msgr2=0x7f3934122020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 1 -- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 msgr2=0x7f3934122020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 0x7f3934122020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:11:56.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f39337fe640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 0x7f3934122020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3932ffd640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f393409e960 0x7f393411fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient(hunting): handle_auth_done global_id 22424 payload 824 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 1 -- 172.21.15.138:0/1558748622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f393409e960 msgr2=0x7f393411fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f393409e960 0x7f393411fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 1 -- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 msgr2=0x7f3934122020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 0x7f3934122020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient: _finish_hunting 0 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 1 monclient: found mon.noname-c 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3933fff640 1 -- 172.21.15.138:0/1558748622 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3934073780 con 0x7f39341356f0 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3932ffd640 10 monclient: get_auth_request con 0x7f393409e960 auth_method 0 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.877+0000 7f3932ffd640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f393409e960 0x7f393411fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:11:56.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f3933fff640 1 -- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39341356f0 msgr2=0x7f393411ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f3933fff640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39341356f0 0x7f393411ff80 secure :-1 s=READY pgs=5686 cs=0 l=1 rev1=1 crypto rx=0x7f392400c7b0 tx=0x7f392400cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 client.?.objecter ms_handle_connect 0x7f39341356f0 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 1 -- 172.21.15.138:0/1558748622 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3924016020 con 0x7f39341356f0 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: dump: 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:56.895 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: _finish_auth 0 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 1 -- 172.21.15.138:0/1558748622 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f392400a9f0 con 0x7f39341356f0 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 5 monclient: authenticate success, global_id 22424 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 10 monclient: _renew_subs 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39341229b0 con 0x7f39341356f0 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 1 -- 172.21.15.138:0/1558748622 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392400ade0 con 0x7f39341356f0 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:11:56.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 monclient: dump: 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f3931ffb640 4 set_mon_vals no callback set 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 10 client.22424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 10 monclient: _renew_subs 2024-09-11T05:11:56.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3934123260 con 0x7f39341356f0 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 1 librados: init done 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f393409e420 con 0x7f39341356f0 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 1 -- 172.21.15.138:0/1558748622 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3924012420 con 0x7f39341356f0 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 1 --2- 172.21.15.138:0/1558748622 >> v2:172.21.15.190:6800/1562782189 conn(0x7f392003ced0 0x7f392003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 1 -- 172.21.15.138:0/1558748622 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(198..198 src has 1..198) ==== 5508+0+0 (secure 0 0 0) 0x7f3924050000 con 0x7f39341356f0 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 10 client.22424.objecter ms_dispatch 0x7f3934000ba0 osd_map(198..198 src has 1..198) 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 3 client.22424.objecter handle_osd_map got epochs [198,198] > 0 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 3 client.22424.objecter handle_osd_map decoding full epoch 198 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f391bfff640 20 client.22424.objecter dump_active .. 0 homeless 2024-09-11T05:11:56.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f39337fe640 1 --2- 172.21.15.138:0/1558748622 >> v2:172.21.15.190:6800/1562782189 conn(0x7f392003ced0 0x7f392003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:11:56.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.881+0000 7f39337fe640 10 monclient: get_auth_request con 0x7f392003ced0 auth_method 0 2024-09-11T05:11:56.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.885+0000 7f39337fe640 1 --2- 172.21.15.138:0/1558748622 >> v2:172.21.15.190:6800/1562782189 conn(0x7f392003ced0 0x7f392003f390 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f3934121fb0 tx=0x7f392c00c2e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:11:56.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.885+0000 7f391bfff640 10 client.22424.objecter ms_handle_connect 0x7f392003ced0 2024-09-11T05:11:56.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.885+0000 7f391bfff640 1 -- 172.21.15.138:0/1558748622 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3924020020 con 0x7f39341356f0 2024-09-11T05:11:56.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.889+0000 7f391bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:11:56.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:56.889+0000 7f391bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:11:57.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.121+0000 7f393a148640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:11:57.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.121+0000 7f393a148640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:57.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.121+0000 7f393a148640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:11:57.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.121+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f39341214c0 con 0x7f39341356f0 2024-09-11T05:11:57.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.121+0000 7f391bfff640 1 -- 172.21.15.138:0/1558748622 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f3924011c60 con 0x7f39341356f0 2024-09-11T05:11:57.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.121+0000 7f391bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:11:57.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.121+0000 7f391bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:11:57.122 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":112,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:11:57.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.121+0000 7f393a148640 10 librados: watch_flush enter 2024-09-11T05:11:57.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 10 librados: watch_flush exit 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 20 client.22424.objecter shutdown clearing up homeless session... 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 10 client.22424.objecter successfully canceled tick 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 >> v2:172.21.15.190:6800/1562782189 conn(0x7f392003ced0 msgr2=0x7f392003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 --2- 172.21.15.138:0/1558748622 >> v2:172.21.15.190:6800/1562782189 conn(0x7f392003ced0 0x7f392003f390 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f3934121fb0 tx=0x7f392c00c2e0 comp rx=0 tx=0).stop 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 10 monclient: shutdown 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39341356f0 msgr2=0x7f393411ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39341356f0 0x7f393411ff80 secure :-1 s=READY pgs=5686 cs=0 l=1 rev1=1 crypto rx=0x7f392400c7b0 tx=0x7f392400cc70 comp rx=0 tx=0).stop 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 shutdown_connections 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 --2- 172.21.15.138:0/1558748622 >> v2:172.21.15.190:6800/1562782189 conn(0x7f392003ced0 0x7f392003f390 unknown :-1 s=CLOSED pgs=3355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39341356f0 0x7f393411ff80 unknown :-1 s=CLOSED pgs=5686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f393409e960 0x7f393411fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:57.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 --2- 172.21.15.138:0/1558748622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f393406e0e0 0x7f3934122020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:11:57.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 >> 172.21.15.138:0/1558748622 conn(0x7f3934063640 msgr2=0x7f39340a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:11:57.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 shutdown_connections 2024-09-11T05:11:57.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 -- 172.21.15.138:0/1558748622 wait complete. 2024-09-11T05:11:57.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:11:57.125+0000 7f393a148640 1 librados: shutdown 2024-09-11T05:12:00.140 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient: get_monmap_and_config 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient: build_initial_monmap 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient: monmap: 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:00.271284+0000 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:00.271284+0000 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:00.270 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:00.271 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:00.271 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:00.271 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:00.271 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:00.271 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:00.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 1 Processor -- start 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 1 -- start start 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient: init 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fea140a5b40 0x7fea140a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient(hunting): picked mon.noname-c con 0x7fea140a5b40 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea140a7130 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea1409d840 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1409d490 con 0x7fea1409d840 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea14139720 con 0x7fea140a7130 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fea1409e110 con 0x7fea140a5b40 2024-09-11T05:12:00.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea1b327640 10 monclient(hunting): authenticate will time out at 1203.707249s 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea198c4640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fea1409d840 0x7fea14139010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33590/0 (socket says 172.21.15.138:33590) 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea198c4640 1 -- 172.21.15.138:0/1180903936 learned_addr learned my addr 172.21.15.138:0/1180903936 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea190c3640 1 --2- 172.21.15.138:0/1180903936 >> v2:172.21.15.190:3300/0 conn(0x7fea140a5b40 0x7fea140a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea190c3640 10 monclient(hunting): get_auth_request con 0x7fea140a5b40 auth_method 0 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea190c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea190c3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea190c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea190c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.269+0000 7fea190c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 10 monclient(hunting): handle_auth_done global_id 20563 payload 274 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 1 -- 172.21.15.138:0/1180903936 >> v1:172.21.15.190:6789/0 conn(0x7fea140a7130 legacy=0x7fea14138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 1 -- 172.21.15.138:0/1180903936 >> v1:172.21.15.138:6789/0 conn(0x7fea1409d840 legacy=0x7fea14139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:00.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 10 monclient: _finish_hunting 0 2024-09-11T05:12:00.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 1 monclient: found mon.noname-c 2024-09-11T05:12:00.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:00.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:12:00.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.273+0000 7fea190c3640 1 -- 172.21.15.138:0/1180903936 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea14070580 con 0x7fea140a5b40 2024-09-11T05:12:00.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea190c3640 1 -- 172.21.15.138:0/1180903936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea140a5b40 msgr2=0x7fea140a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:00.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea190c3640 1 --2- 172.21.15.138:0/1180903936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea140a5b40 0x7fea140a5f20 secure :-1 s=READY pgs=5772 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009d90 tx=0x7fea0c01c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af3e82eea514a563 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:00.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 1 -- 172.21.15.138:0/1180903936 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c0b2070 con 0x7fea140a5b40 2024-09-11T05:12:00.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:00.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:00.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: dump: 2024-09-11T05:12:00.279 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:00.279 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: _finish_auth 0 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:00.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 1 -- 172.21.15.138:0/1180903936 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea0c0a5860 con 0x7fea140a5b40 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 5 monclient: authenticate success, global_id 20563 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 1 -- 172.21.15.138:0/1180903936 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c0a5c30 con 0x7fea140a5b40 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 4 set_mon_vals no callback set 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea0bfff640 10 monclient: dump: 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:00.281 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:00.282 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:00.282 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:00.282 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:00.282 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:00.282 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:00.282 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:00.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient: shutdown 2024-09-11T05:12:00.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- 172.21.15.138:0/1180903936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea140a5b40 msgr2=0x7fea140a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 --2- 172.21.15.138:0/1180903936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea140a5b40 0x7fea140a5f20 secure :-1 s=READY pgs=5772 cs=0 l=1 rev1=1 crypto rx=0x7fea0c009d90 tx=0x7fea0c01c830 comp rx=0 tx=0).stop 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- 172.21.15.138:0/1180903936 shutdown_connections 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 --2- 172.21.15.138:0/1180903936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea140a5b40 0x7fea140a5f20 unknown :-1 s=CLOSED pgs=5772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- 172.21.15.138:0/1180903936 >> 172.21.15.138:0/1180903936 conn(0x7fea140aba10 msgr2=0x7fea140a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- 172.21.15.138:0/1180903936 shutdown_connections 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- 172.21.15.138:0/1180903936 wait complete. 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient: build_initial_monmap 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient: monmap: 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:00.288 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 librados: starting msgr at 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 librados: starting objecter 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 Processor -- start 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- start start 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 librados: setting wanted keys 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 librados: calling monclient init 2024-09-11T05:12:00.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient: init 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1409d840 0x7fea14140b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient(hunting): picked mon.noname-b con 0x7fea1409d840 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 0x7fea141d1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient(hunting): picked mon.noname-a con 0x7fea140a5b40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea140a7130 0x7fea1413e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient(hunting): picked mon.noname-c con 0x7fea140a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fea14144920 con 0x7fea140a5b40 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea141447a0 con 0x7fea1409d840 2024-09-11T05:12:00.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fea14144aa0 con 0x7fea140a7130 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea1b327640 10 monclient(hunting): authenticate will time out at 1203.715796s 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea188c2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 0x7fea141d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea188c2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 0x7fea141d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35830/0 (socket says 172.21.15.138:35830) 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea188c2640 1 -- 172.21.15.138:0/2667636410 learned_addr learned my addr 172.21.15.138:0/2667636410 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea188c2640 10 monclient(hunting): get_auth_request con 0x7fea140a5b40 auth_method 0 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea188c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea188c2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea188c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.277+0000 7fea198c4640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea140a7130 0x7fea1413e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient(hunting): get_auth_request con 0x7fea140a7130 auth_method 0 2024-09-11T05:12:00.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea188c2640 1 -- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 msgr2=0x7fea141d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea188c2640 1 -- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 msgr2=0x7fea141d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea188c2640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 0x7fea141d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea188c2640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 0x7fea141d1270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea190c3640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1409d840 0x7fea14140b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea190c3640 10 monclient(hunting): get_auth_request con 0x7fea1409d840 auth_method 0 2024-09-11T05:12:00.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea190c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea190c3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea190c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea190c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea190c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea190c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient(hunting): handle_auth_done global_id 22433 payload 824 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 1 -- 172.21.15.138:0/2667636410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1409d840 msgr2=0x7fea14140b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1409d840 0x7fea14140b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 1 -- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 msgr2=0x7fea141d1270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 0x7fea141d1270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient: _finish_hunting 0 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 1 monclient: found mon.noname-c 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:00.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 1 -- 172.21.15.138:0/2667636410 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea14121fd0 con 0x7fea140a7130 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 1 -- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea140a7130 msgr2=0x7fea1413e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea198c4640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea140a7130 0x7fea1413e5e0 secure :-1 s=READY pgs=5687 cs=0 l=1 rev1=1 crypto rx=0x7fea1000c440 tx=0x7fea1000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 client.?.objecter ms_handle_connect 0x7fea140a7130 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 1 -- 172.21.15.138:0/2667636410 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea10017070 con 0x7fea140a7130 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: dump: 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:00.294 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: _finish_auth 0 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 1 -- 172.21.15.138:0/2667636410 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea100070d0 con 0x7fea140a7130 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 5 monclient: authenticate success, global_id 22433 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 10 monclient: _renew_subs 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea141219a0 con 0x7fea140a7130 2024-09-11T05:12:00.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 1 -- 172.21.15.138:0/2667636410 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea100074c0 con 0x7fea140a7130 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 10 monclient: dump: 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:00.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea0b7fe640 4 set_mon_vals no callback set 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 10 client.22433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 10 monclient: _renew_subs 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea141c9b30 con 0x7fea140a7130 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 1 librados: init done 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea140a5f20 con 0x7fea140a7130 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.281+0000 7fea09ffb640 1 -- 172.21.15.138:0/2667636410 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fea1001b3b0 con 0x7fea140a7130 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea09ffb640 1 --2- 172.21.15.138:0/2667636410 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe9f803cea0 0x7fe9f803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea09ffb640 1 -- 172.21.15.138:0/2667636410 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(201..201 src has 1..201) ==== 6683+0+0 (secure 0 0 0) 0x7fea1004f5c0 con 0x7fea140a7130 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea09ffb640 10 client.22433.objecter ms_dispatch 0x7fea14000ba0 osd_map(201..201 src has 1..201) 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea09ffb640 3 client.22433.objecter handle_osd_map got epochs [201,201] > 0 2024-09-11T05:12:00.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea09ffb640 3 client.22433.objecter handle_osd_map decoding full epoch 201 2024-09-11T05:12:00.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea09ffb640 20 client.22433.objecter dump_active .. 0 homeless 2024-09-11T05:12:00.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea190c3640 1 --2- 172.21.15.138:0/2667636410 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe9f803cea0 0x7fe9f803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:00.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea190c3640 10 monclient: get_auth_request con 0x7fe9f803cea0 auth_method 0 2024-09-11T05:12:00.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea190c3640 1 --2- 172.21.15.138:0/2667636410 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe9f803cea0 0x7fe9f803f360 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0ac000 tx=0x7fea0c0a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:00.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.285+0000 7fea09ffb640 10 client.22433.objecter ms_handle_connect 0x7fe9f803cea0 2024-09-11T05:12:00.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.289+0000 7fea09ffb640 1 -- 172.21.15.138:0/2667636410 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fea1001a360 con 0x7fea140a7130 2024-09-11T05:12:00.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.289+0000 7fea09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:00.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.289+0000 7fea09ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:00.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:12:00.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:00.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:00.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea14069fa0 con 0x7fea140a7130 2024-09-11T05:12:00.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea09ffb640 1 -- 172.21.15.138:0/2667636410 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fea100192c0 con 0x7fea140a7130 2024-09-11T05:12:00.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:00.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea09ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:00.538 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":115,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:12:00.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 10 librados: watch_flush enter 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 10 librados: watch_flush exit 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 20 client.22433.objecter shutdown clearing up homeless session... 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 10 client.22433.objecter successfully canceled tick 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe9f803cea0 msgr2=0x7fe9f803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 1 --2- 172.21.15.138:0/2667636410 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe9f803cea0 0x7fe9f803f360 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0ac000 tx=0x7fea0c0a7070 comp rx=0 tx=0).stop 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 10 monclient: shutdown 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea140a7130 msgr2=0x7fea1413e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.537+0000 7fea1b327640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea140a7130 0x7fea1413e5e0 secure :-1 s=READY pgs=5687 cs=0 l=1 rev1=1 crypto rx=0x7fea1000c440 tx=0x7fea1000c900 comp rx=0 tx=0).stop 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 shutdown_connections 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 --2- 172.21.15.138:0/2667636410 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe9f803cea0 0x7fe9f803f360 unknown :-1 s=CLOSED pgs=3357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:00.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea140a7130 0x7fea1413e5e0 unknown :-1 s=CLOSED pgs=5687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:00.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea140a5b40 0x7fea141d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:00.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 --2- 172.21.15.138:0/2667636410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1409d840 0x7fea14140b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:00.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 >> 172.21.15.138:0/2667636410 conn(0x7fea140aba10 msgr2=0x7fea140a92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:00.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 shutdown_connections 2024-09-11T05:12:00.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 -- 172.21.15.138:0/2667636410 wait complete. 2024-09-11T05:12:00.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:00.541+0000 7fea1b327640 1 librados: shutdown 2024-09-11T05:12:03.557 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:12:03.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient: get_monmap_and_config 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient: build_initial_monmap 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient: monmap: 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:03.687015+0000 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:03.687015+0000 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:03.686 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:03.687 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:03.687 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:03.687 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:03.687 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:03.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 1 Processor -- start 2024-09-11T05:12:03.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 1 -- start start 2024-09-11T05:12:03.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient: init 2024-09-11T05:12:03.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ea80a0a90 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea809dc20 0x7f8ea80a2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea809dc20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f8ea80a2bb0 0x7f8ea813bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea80a2bb0 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ea809efa0 con 0x7f8ea80a0a90 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f8ea80a37b0 con 0x7f8ea80a2bb0 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8ea809ecc0 con 0x7f8ea809dc20 2024-09-11T05:12:03.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8ead61c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f8ea80a0a90 0x7f8ea80a0e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33596/0 (socket says 172.21.15.138:33596) 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8ead61c640 1 -- 172.21.15.138:0/1352095489 learned_addr learned my addr 172.21.15.138:0/1352095489 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eaf880640 10 monclient(hunting): authenticate will time out at 1207.123367s 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 1 --2- 172.21.15.138:0/1352095489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea809dc20 0x7f8ea80a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient(hunting): get_auth_request con 0x7f8ea809dc20 auth_method 0 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eade1d640 1 --2- 172.21.15.138:0/1352095489 >> v2:172.21.15.190:3300/0 conn(0x7f8ea80a2bb0 0x7f8ea813bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:03.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eade1d640 10 monclient(hunting): get_auth_request con 0x7f8ea80a2bb0 auth_method 0 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eade1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eade1d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eade1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient(hunting): handle_auth_done global_id 22439 payload 274 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 1 -- 172.21.15.138:0/1352095489 >> v2:172.21.15.190:3300/0 conn(0x7f8ea80a2bb0 msgr2=0x7f8ea813bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 1 --2- 172.21.15.138:0/1352095489 >> v2:172.21.15.190:3300/0 conn(0x7f8ea80a2bb0 0x7f8ea813bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 1 -- 172.21.15.138:0/1352095489 >> v1:172.21.15.138:6789/0 conn(0x7f8ea80a0a90 legacy=0x7f8ea80a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient: _finish_hunting 0 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 1 monclient: found mon.noname-b 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:03.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.685+0000 7f8eace1b640 1 -- 172.21.15.138:0/1352095489 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea806e340 con 0x7f8ea809dc20 2024-09-11T05:12:03.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.689+0000 7f8eace1b640 1 -- 172.21.15.138:0/1352095489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea809dc20 msgr2=0x7f8ea80a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:03.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.689+0000 7f8eace1b640 1 --2- 172.21.15.138:0/1352095489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea809dc20 0x7f8ea80a2670 secure :-1 s=READY pgs=5691 cs=0 l=1 rev1=1 crypto rx=0x7f8e980097c0 tx=0x7f8e9801c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a0a37c861816e79 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:03.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 1 -- 172.21.15.138:0/1352095489 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e980b0070 con 0x7f8ea809dc20 2024-09-11T05:12:03.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:03.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:03.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: dump: 2024-09-11T05:12:03.699 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:03.699 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: _finish_auth 0 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 1 -- 172.21.15.138:0/1352095489 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e980a9710 con 0x7f8ea809dc20 2024-09-11T05:12:03.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 1 -- 172.21.15.138:0/1352095489 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e980a9ae0 con 0x7f8ea809dc20 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 5 monclient: authenticate success, global_id 22439 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 4 set_mon_vals no callback set 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8e9ffff640 10 monclient: dump: 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:03.701 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient: shutdown 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- 172.21.15.138:0/1352095489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea809dc20 msgr2=0x7f8ea80a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 --2- 172.21.15.138:0/1352095489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea809dc20 0x7f8ea80a2670 secure :-1 s=READY pgs=5691 cs=0 l=1 rev1=1 crypto rx=0x7f8e980097c0 tx=0x7f8e9801c180 comp rx=0 tx=0).stop 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- 172.21.15.138:0/1352095489 shutdown_connections 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 --2- 172.21.15.138:0/1352095489 >> v2:172.21.15.190:3300/0 conn(0x7f8ea80a2bb0 0x7f8ea813bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 --2- 172.21.15.138:0/1352095489 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea809dc20 0x7f8ea80a2670 unknown :-1 s=CLOSED pgs=5691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- 172.21.15.138:0/1352095489 >> 172.21.15.138:0/1352095489 conn(0x7f8ea8063640 msgr2=0x7f8ea8063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- 172.21.15.138:0/1352095489 shutdown_connections 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- 172.21.15.138:0/1352095489 wait complete. 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient: build_initial_monmap 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient: monmap: 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:03.703 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 librados: starting msgr at 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 librados: starting objecter 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 Processor -- start 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- start start 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 librados: setting wanted keys 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 librados: calling monclient init 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient: init 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea809dc20 0x7f8ea8134280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea809dc20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:03.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea80a0a90 0x7f8ea81d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea80a0a90 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ea81d10d0 0x7f8ea81cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea81d10d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea813ccd0 con 0x7f8ea81d10d0 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea813cb50 con 0x7f8ea809dc20 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8ea813ce50 con 0x7f8ea80a0a90 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eaf880640 10 monclient(hunting): authenticate will time out at 1207.130596s 2024-09-11T05:12:03.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ea81d10d0 0x7f8ea81cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ea81d10d0 0x7f8ea81cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35846/0 (socket says 172.21.15.138:35846) 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 -- 172.21.15.138:0/2727768472 learned_addr learned my addr 172.21.15.138:0/2727768472 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient(hunting): get_auth_request con 0x7f8ea81d10d0 auth_method 0 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eace1b640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea80a0a90 0x7f8ea81d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eace1b640 10 monclient(hunting): get_auth_request con 0x7f8ea80a0a90 auth_method 0 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eace1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eace1b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eace1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:03.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8ead61c640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea809dc20 0x7f8ea8134280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eace1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eace1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eace1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8ead61c640 10 monclient(hunting): get_auth_request con 0x7f8ea809dc20 auth_method 0 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8ead61c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8ead61c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8ead61c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient(hunting): handle_auth_done global_id 64107 payload 824 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 -- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea80a0a90 msgr2=0x7f8ea81d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea80a0a90 0x7f8ea81d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 -- 172.21.15.138:0/2727768472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea809dc20 msgr2=0x7f8ea8134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:03.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea809dc20 0x7f8ea8134280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient: _finish_hunting 0 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 monclient: found mon.noname-a 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8eade1d640 1 -- 172.21.15.138:0/2727768472 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea80723d0 con 0x7f8ea81d10d0 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.693+0000 7f8ead61c640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea809dc20 0x7f8ea8134280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eade1d640 1 -- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ea81d10d0 msgr2=0x7f8ea81cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eade1d640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ea81d10d0 0x7f8ea81cd1d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0007d80 tx=0x7f8ea000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 client.?.objecter ms_handle_connect 0x7f8ea81d10d0 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 1 -- 172.21.15.138:0/2727768472 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0014620 con 0x7f8ea81d10d0 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:03.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: dump: 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:03.709 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: _finish_auth 0 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: _check_auth_tickets 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 1 -- 172.21.15.138:0/2727768472 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea0019070 con 0x7f8ea81d10d0 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 5 monclient: authenticate success, global_id 64107 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 10 monclient: _renew_subs 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea8070bd0 con 0x7f8ea81d10d0 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 1 -- 172.21.15.138:0/2727768472 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0014a00 con 0x7f8ea81d10d0 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 monclient: dump: 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:03.710 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9f7fe640 4 set_mon_vals no callback set 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 10 client.64107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 10 monclient: _renew_subs 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:03.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea8134c40 con 0x7f8ea81d10d0 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 1 librados: init done 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea80a2670 con 0x7f8ea81d10d0 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 1 -- 172.21.15.138:0/2727768472 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f8ea0014ba0 con 0x7f8ea81d10d0 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 1 --2- 172.21.15.138:0/2727768472 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8e8c03cea0 0x7f8e8c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 1 -- 172.21.15.138:0/2727768472 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5489+0+0 (secure 0 0 0) 0x7f8ea0050280 con 0x7f8ea81d10d0 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 10 client.64107.objecter ms_dispatch 0x7f8ea8000ba0 osd_map(205..205 src has 1..205) 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 3 client.64107.objecter handle_osd_map got epochs [205,205] > 0 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 3 client.64107.objecter handle_osd_map decoding full epoch 205 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8e9dffb640 20 client.64107.objecter dump_active .. 0 homeless 2024-09-11T05:12:03.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8ead61c640 1 --2- 172.21.15.138:0/2727768472 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8e8c03cea0 0x7f8e8c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:03.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.697+0000 7f8ead61c640 10 monclient: get_auth_request con 0x7f8e8c03cea0 auth_method 0 2024-09-11T05:12:03.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.701+0000 7f8ead61c640 1 --2- 172.21.15.138:0/2727768472 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8e8c03cea0 0x7f8e8c03f360 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4007fd0 tx=0x7f8ea4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:03.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.701+0000 7f8e9dffb640 10 client.64107.objecter ms_handle_connect 0x7f8e8c03cea0 2024-09-11T05:12:03.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.701+0000 7f8e9dffb640 1 -- 172.21.15.138:0/2727768472 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f8ea004e9b0 con 0x7f8ea81d10d0 2024-09-11T05:12:03.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.701+0000 7f8e9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:03.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.701+0000 7f8e9dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:03.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.937+0000 7f8eaf880640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:12:03.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.937+0000 7f8eaf880640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:03.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.937+0000 7f8eaf880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:03.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.937+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8ea8072150 con 0x7f8ea81d10d0 2024-09-11T05:12:03.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.937+0000 7f8e9dffb640 1 -- 172.21.15.138:0/2727768472 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f8ea0012cf0 con 0x7f8ea81d10d0 2024-09-11T05:12:03.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.937+0000 7f8e9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:03.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.937+0000 7f8e9dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:03.940 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:12:03.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 10 librados: watch_flush enter 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 10 librados: watch_flush exit 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 20 client.64107.objecter shutdown clearing up homeless session... 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 10 client.64107.objecter successfully canceled tick 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8e8c03cea0 msgr2=0x7f8e8c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 --2- 172.21.15.138:0/2727768472 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8e8c03cea0 0x7f8e8c03f360 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4007fd0 tx=0x7f8ea4009040 comp rx=0 tx=0).stop 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 10 monclient: shutdown 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ea81d10d0 msgr2=0x7f8ea81cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ea81d10d0 0x7f8ea81cd1d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0007d80 tx=0x7f8ea000e4b0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 shutdown_connections 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 --2- 172.21.15.138:0/2727768472 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8e8c03cea0 0x7f8e8c03f360 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8ea81d10d0 0x7f8ea81cd1d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8ea80a0a90 0x7f8ea81d0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 --2- 172.21.15.138:0/2727768472 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ea809dc20 0x7f8ea8134280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:03.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 >> 172.21.15.138:0/2727768472 conn(0x7f8ea8063640 msgr2=0x7f8ea80a5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:03.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 shutdown_connections 2024-09-11T05:12:03.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 -- 172.21.15.138:0/2727768472 wait complete. 2024-09-11T05:12:03.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:03.941+0000 7f8eaf880640 1 librados: shutdown 2024-09-11T05:12:03.959 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:12:03.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:12:03.959 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:12:04.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient: get_monmap_and_config 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient: build_initial_monmap 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient: monmap: 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:04.073926+0000 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:04.073926+0000 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.073 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:04.074 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:04.074 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:04.074 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:04.074 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:04.074 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 1 Processor -- start 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 1 -- start start 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient: init 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac40a32c0 0x7f5ac409e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac40a32c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5ac409e960 0x7f5ac409ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac409e960 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5ac41356f0 0x7f5ac4135ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac41356f0 addr v2:172.21.15.138:3300/0 2024-09-11T05:12:04.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac409fe70 con 0x7f5ac41356f0 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac40a1920 con 0x7f5ac409e960 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac40a36a0 con 0x7f5ac40a32c0 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac9fb8640 10 monclient(hunting): authenticate will time out at 1207.510506s 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac40a32c0 0x7f5ac409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5ac41356f0 0x7f5ac4135ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac40a32c0 0x7f5ac409e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44156/0 (socket says 172.21.15.138:44156) 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 -- 172.21.15.138:0/2764480147 learned_addr learned my addr 172.21.15.138:0/2764480147 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5ac41356f0 0x7f5ac4135ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35860/0 (socket says 172.21.15.138:35860) 2024-09-11T05:12:04.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient(hunting): get_auth_request con 0x7f5ac40a32c0 auth_method 0 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 10 monclient(hunting): get_auth_request con 0x7f5ac41356f0 auth_method 0 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5abaffd640 1 --2- 172.21.15.138:0/2764480147 >> v2:172.21.15.190:3300/0 conn(0x7f5ac409e960 0x7f5ac409ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5abaffd640 10 monclient(hunting): get_auth_request con 0x7f5ac409e960 auth_method 0 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5abaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5abaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient(hunting): handle_auth_done global_id 22445 payload 274 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 -- 172.21.15.138:0/2764480147 >> v2:172.21.15.190:3300/0 conn(0x7f5ac409e960 msgr2=0x7f5ac409ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 --2- 172.21.15.138:0/2764480147 >> v2:172.21.15.190:3300/0 conn(0x7f5ac409e960 0x7f5ac409ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 -- 172.21.15.138:0/2764480147 >> v2:172.21.15.138:3300/0 conn(0x7f5ac41356f0 msgr2=0x7f5ac4135ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 --2- 172.21.15.138:0/2764480147 >> v2:172.21.15.138:3300/0 conn(0x7f5ac41356f0 0x7f5ac4135ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 monclient: found mon.noname-b 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:04.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac37fe640 1 -- 172.21.15.138:0/2764480147 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac40a1b50 con 0x7f5ac40a32c0 2024-09-11T05:12:04.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5ac3fff640 1 --2- 172.21.15.138:0/2764480147 >> v2:172.21.15.138:3300/0 conn(0x7f5ac41356f0 0x7f5ac4135ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:04.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.073+0000 7f5abaffd640 1 --2- 172.21.15.138:0/2764480147 >> v2:172.21.15.190:3300/0 conn(0x7f5ac409e960 0x7f5ac409ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:12:04.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac37fe640 1 -- 172.21.15.138:0/2764480147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac40a32c0 msgr2=0x7f5ac409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac37fe640 1 --2- 172.21.15.138:0/2764480147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac40a32c0 0x7f5ac409e420 secure :-1 s=READY pgs=5692 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40a3950 tx=0x7f5ab401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36bbd1fb6ebf20a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 1 -- 172.21.15.138:0/2764480147 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40b0070 con 0x7f5ac40a32c0 2024-09-11T05:12:04.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.087 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: dump: 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: _finish_auth 0 2024-09-11T05:12:04.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 1 -- 172.21.15.138:0/2764480147 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab40ab600 con 0x7f5ac40a32c0 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac9fb8640 5 monclient: authenticate success, global_id 22445 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac9fb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac9fb8640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 1 -- 172.21.15.138:0/2764480147 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40ab9d0 con 0x7f5ac40a32c0 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac9fb8640 4 set_mon_vals no callback set 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:04.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.077+0000 7f5ac2ffd640 10 monclient: dump: 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient: shutdown 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/2764480147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac40a32c0 msgr2=0x7f5ac409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 --2- 172.21.15.138:0/2764480147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac40a32c0 0x7f5ac409e420 secure :-1 s=READY pgs=5692 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40a3950 tx=0x7f5ab401c410 comp rx=0 tx=0).stop 2024-09-11T05:12:04.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/2764480147 shutdown_connections 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 --2- 172.21.15.138:0/2764480147 >> v2:172.21.15.138:3300/0 conn(0x7f5ac41356f0 0x7f5ac4135ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 --2- 172.21.15.138:0/2764480147 >> v2:172.21.15.190:3300/0 conn(0x7f5ac409e960 0x7f5ac409ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 --2- 172.21.15.138:0/2764480147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac40a32c0 0x7f5ac409e420 unknown :-1 s=CLOSED pgs=5692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/2764480147 >> 172.21.15.138:0/2764480147 conn(0x7f5ac4063640 msgr2=0x7f5ac4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/2764480147 shutdown_connections 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/2764480147 wait complete. 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient: build_initial_monmap 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient: monmap: 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:04.091 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 librados: starting msgr at 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 librados: starting objecter 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 Processor -- start 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- start start 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 librados: setting wanted keys 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 librados: calling monclient init 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient: init 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac409e960 0x7f5ac4130fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac409e960 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ac40a32c0 0x7f5ac406f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac40a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac41356f0 0x7f5ac406f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac41356f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac406e1f0 con 0x7f5ac409e960 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac409f900 con 0x7f5ac40a32c0 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac40a12d0 con 0x7f5ac41356f0 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac9fb8640 10 monclient(hunting): authenticate will time out at 1207.517924s 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac409e960 0x7f5ac4130fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac41356f0 0x7f5ac406f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac409e960 0x7f5ac4130fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35872/0 (socket says 172.21.15.138:35872) 2024-09-11T05:12:04.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 -- 172.21.15.138:0/1855531692 learned_addr learned my addr 172.21.15.138:0/1855531692 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient(hunting): get_auth_request con 0x7f5ac409e960 auth_method 0 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac3fff640 10 monclient(hunting): get_auth_request con 0x7f5ac41356f0 auth_method 0 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5abaffd640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ac40a32c0 0x7f5ac406f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient(hunting): handle_auth_done global_id 64113 payload 824 2024-09-11T05:12:04.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac41356f0 msgr2=0x7f5ac406f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ac41356f0 0x7f5ac406f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ac40a32c0 msgr2=0x7f5ac406f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ac40a32c0 0x7f5ac406f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 monclient: found mon.noname-a 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.081+0000 7f5ac37fe640 1 -- 172.21.15.138:0/1855531692 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac4139560 con 0x7f5ac409e960 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac37fe640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac409e960 msgr2=0x7f5ac4130fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac37fe640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac409e960 0x7f5ac4130fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40ae040 tx=0x7f5ab401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 client.?.objecter ms_handle_connect 0x7f5ac409e960 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40ba020 con 0x7f5ac409e960 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:04.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: dump: 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: _finish_auth 0 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab40abe10 con 0x7f5ac409e960 2024-09-11T05:12:04.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 5 monclient: authenticate success, global_id 64113 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 10 monclient: _renew_subs 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac41314e0 con 0x7f5ac409e960 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40b0040 con 0x7f5ac409e960 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 monclient: dump: 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.098 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac27fc640 4 set_mon_vals no callback set 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 10 client.64113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 10 monclient: _renew_subs 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac409d310 con 0x7f5ac409e960 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 1 librados: init done 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac406fd90 0x7f5ac40714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac4070170 con 0x7f5ac406fd90 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac406fd90 0x7f5ac40714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 10 monclient: get_auth_request con 0x7f5ac406fd90 auth_method 0 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5ab40b3030 con 0x7f5ac409e960 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 10 monclient(hunting): handle_auth_done global_id 64116 payload 824 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5abaffd640 1 -- 172.21.15.138:0/1855531692 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ac409e110 con 0x7f5ac406fd90 2024-09-11T05:12:04.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 1 --2- 172.21.15.138:0/1855531692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5aa403cef0 0x7f5aa403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5489+0+0 (secure 0 0 0) 0x7f5ab40e9bf0 con 0x7f5ac409e960 2024-09-11T05:12:04.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 10 client.64113.objecter ms_dispatch 0x7f5ac4000ba0 osd_map(205..205 src has 1..205) 2024-09-11T05:12:04.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 3 client.64113.objecter handle_osd_map got epochs [205,205] > 0 2024-09-11T05:12:04.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 3 client.64113.objecter handle_osd_map decoding full epoch 205 2024-09-11T05:12:04.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac0ff9640 20 client.64113.objecter dump_active .. 0 homeless 2024-09-11T05:12:04.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac3fff640 1 --2- 172.21.15.138:0/1855531692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5aa403cef0 0x7f5aa403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.085+0000 7f5ac3fff640 10 monclient: get_auth_request con 0x7f5aa403cef0 auth_method 0 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 10 client.64113.objecter ms_handle_connect 0x7f5ac406fd90 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5abaffd640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac406fd90 msgr2=0x7f5ac40714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 10 client.64113.objecter resend_mon_ops 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5abaffd640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac406fd90 0x7f5ac40714f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40a1d90 tx=0x7f5aac00e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac017070 con 0x7f5ac406fd90 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 10 monclient: dump: 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.104 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5aac013900 con 0x7f5ac406fd90 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac406fd90 msgr2=0x7f5ac40714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac406fd90 0x7f5ac40714f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40a1d90 tx=0x7f5aac00e900 comp rx=0 tx=0).stop 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac406fd90 msgr2=0x7f5ac40714f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:04.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac406fd90 0x7f5ac40714f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac0ff9640 10 client.64113.objecter ms_handle_connect 0x7f5aa403cef0 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.089+0000 7f5ac3fff640 1 --2- 172.21.15.138:0/1855531692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5aa403cef0 0x7f5aa403f3b0 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7f5abc011420 tx=0x7f5abc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5ac9fb8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5ac9fb8640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac4071a30 0x7f5ac4071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5ac9fb8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac409e420 con 0x7f5ac4071a30 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac4071a30 0x7f5ac4071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 10 monclient: get_auth_request con 0x7f5ac4071a30 auth_method 0 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 10 monclient(hunting): handle_auth_done global_id 64119 payload 824 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.093+0000 7f5abaffd640 1 -- 172.21.15.138:0/1855531692 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ac413b4b0 con 0x7f5ac4071a30 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 10 client.64113.objecter ms_handle_connect 0x7f5ac4071a30 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 10 client.64113.objecter resend_mon_ops 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5abaffd640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac4071a30 msgr2=0x7f5ac4071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5abaffd640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac4071a30 0x7f5ac4071e40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5aac019000 tx=0x7f5aac004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00f040 con 0x7f5ac4071a30 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 10 monclient: dump: 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.107 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f5aac012780 con 0x7f5ac4071a30 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac4071a30 msgr2=0x7f5ac4071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac4071a30 0x7f5ac4071e40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5aac019000 tx=0x7f5aac004740 comp rx=0 tx=0).stop 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac4071a30 msgr2=0x7f5ac4071e40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:04.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac0ff9640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac4071a30 0x7f5ac4071e40 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5aac019000 tx=0x7f5aac004740 comp rx=0 tx=0).stop 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 20, 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 27727, 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:12:04.109 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.110 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.111 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:04.112 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.113 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.114 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:12:04.115 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:12:04.116 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.118 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.119 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.120 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:12:04.121 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:12:04.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 10 librados: watch_flush enter 2024-09-11T05:12:04.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 10 librados: watch_flush exit 2024-09-11T05:12:04.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 20 client.64113.objecter shutdown clearing up homeless session... 2024-09-11T05:12:04.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 10 client.64113.objecter successfully canceled tick 2024-09-11T05:12:04.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5aa403cef0 msgr2=0x7f5aa403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 --2- 172.21.15.138:0/1855531692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5aa403cef0 0x7f5aa403f3b0 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7f5abc011420 tx=0x7f5abc00b040 comp rx=0 tx=0).stop 2024-09-11T05:12:04.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 10 monclient: shutdown 2024-09-11T05:12:04.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac409e960 msgr2=0x7f5ac4130fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 --2- 172.21.15.138:0/1855531692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ac409e960 0x7f5ac4130fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40ae040 tx=0x7f5ab401acb0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac3fff640 1 -- 172.21.15.138:0/1855531692 reap_dead start 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac3fff640 1 -- 172.21.15.138:0/1855531692 reap_dead start 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 shutdown_connections 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 >> 172.21.15.138:0/1855531692 conn(0x7f5ac4063640 msgr2=0x7f5ac40a5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 shutdown_connections 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 -- 172.21.15.138:0/1855531692 wait complete. 2024-09-11T05:12:04.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.097+0000 7f5ac9fb8640 1 librados: shutdown 2024-09-11T05:12:04.123 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:12:04.124 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:12:04.124 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:12:04.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.229+0000 7fc8b7a0b640 10 monclient: get_monmap_and_config 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.229+0000 7fc8b7a0b640 10 monclient: build_initial_monmap 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.229+0000 7fc8b7a0b640 10 monclient: monmap: 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:04.232930+0000 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:04.232930+0000 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:04.232 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 1 Processor -- start 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 1 -- start start 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient: init 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b00a32c0 0x7fc8b009e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b00a32c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fc8b009e960 0x7fc8b009ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b009e960 addr v2:172.21.15.138:3300/0 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8b01356f0 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8b00a1b50 con 0x7fc8b01356f0 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fc8b009fe70 con 0x7fc8b009e960 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b00a1920 con 0x7fc8b00a32c0 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b7a0b640 10 monclient(hunting): authenticate will time out at 1207.668695s 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fc8b009e960 0x7fc8b009ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b00a32c0 0x7fc8b009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b00a32c0 0x7fc8b009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44168/0 (socket says 172.21.15.138:44168) 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fc8b009e960 0x7fc8b009ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35890/0 (socket says 172.21.15.138:35890) 2024-09-11T05:12:04.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 1 -- 172.21.15.138:0/827013108 learned_addr learned my addr 172.21.15.138:0/827013108 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient(hunting): get_auth_request con 0x7fc8b00a32c0 auth_method 0 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 10 monclient(hunting): get_auth_request con 0x7fc8b009e960 auth_method 0 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient(hunting): handle_auth_done global_id 22451 payload 274 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 1 -- 172.21.15.138:0/827013108 >> v2:172.21.15.138:3300/0 conn(0x7fc8b009e960 msgr2=0x7fc8b009ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 1 --2- 172.21.15.138:0/827013108 >> v2:172.21.15.138:3300/0 conn(0x7fc8b009e960 0x7fc8b009ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 1 -- 172.21.15.138:0/827013108 >> v1:172.21.15.190:6789/0 conn(0x7fc8b01356f0 legacy=0x7fc8b0135ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient: _finish_hunting 0 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 1 monclient: found mon.noname-b 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b57a7640 1 -- 172.21.15.138:0/827013108 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b009f560 con 0x7fc8b00a32c0 2024-09-11T05:12:04.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.233+0000 7fc8b4fa6640 1 --2- 172.21.15.138:0/827013108 >> v2:172.21.15.138:3300/0 conn(0x7fc8b009e960 0x7fc8b009ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:04.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b57a7640 1 -- 172.21.15.138:0/827013108 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b00a32c0 msgr2=0x7fc8b009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b57a7640 1 --2- 172.21.15.138:0/827013108 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b00a32c0 0x7fc8b009e420 secure :-1 s=READY pgs=5693 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009870 tx=0x7fc8ac01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=869a0844bac3c6e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 1 -- 172.21.15.138:0/827013108 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0b0070 con 0x7fc8b00a32c0 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: dump: 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.248 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: _finish_auth 0 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: _check_auth_tickets 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 5 monclient: authenticate success, global_id 22451 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 1 -- 172.21.15.138:0/827013108 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8ac0ab600 con 0x7fc8b00a32c0 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 4 set_mon_vals no callback set 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 1 -- 172.21.15.138:0/827013108 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0a4e60 con 0x7fc8b00a32c0 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8a7fff640 10 monclient: dump: 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 10 monclient: shutdown 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:04.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/827013108 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b00a32c0 msgr2=0x7fc8b009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 --2- 172.21.15.138:0/827013108 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b00a32c0 0x7fc8b009e420 secure :-1 s=READY pgs=5693 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009870 tx=0x7fc8ac01c450 comp rx=0 tx=0).stop 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/827013108 shutdown_connections 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 --2- 172.21.15.138:0/827013108 >> v2:172.21.15.138:3300/0 conn(0x7fc8b009e960 0x7fc8b009ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 --2- 172.21.15.138:0/827013108 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b00a32c0 0x7fc8b009e420 unknown :-1 s=CLOSED pgs=5693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/827013108 >> 172.21.15.138:0/827013108 conn(0x7fc8b0063640 msgr2=0x7fc8b0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/827013108 shutdown_connections 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/827013108 wait complete. 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 10 monclient: build_initial_monmap 2024-09-11T05:12:04.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 10 monclient: monmap: 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 librados: starting msgr at 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 librados: starting objecter 2024-09-11T05:12:04.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 Processor -- start 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 -- start start 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 librados: setting wanted keys 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 1 librados: calling monclient init 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.237+0000 7fc8b7a0b640 10 monclient: init 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b009e960 0x7fc8b013afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b009e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8b00a32c0 0x7fc8b013b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b00a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8b01356f0 0x7fc8b013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b01356f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b00a21b0 con 0x7fc8b01356f0 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b006e280 con 0x7fc8b00a32c0 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b00a2330 con 0x7fc8b009e960 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): authenticate will time out at 1207.676299s 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b009e960 0x7fc8b013afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8b01356f0 0x7fc8b013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b009e960 0x7fc8b013afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44178/0 (socket says 172.21.15.138:44178) 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 -- 172.21.15.138:0/2998367954 learned_addr learned my addr 172.21.15.138:0/2998367954 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient(hunting): get_auth_request con 0x7fc8b009e960 auth_method 0 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 10 monclient(hunting): get_auth_request con 0x7fc8b01356f0 auth_method 0 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b4fa6640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8b00a32c0 0x7fc8b013b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient(hunting): handle_auth_done global_id 22454 payload 824 2024-09-11T05:12:04.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8b00a32c0 msgr2=0x7fc8b013b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8b00a32c0 0x7fc8b013b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8b01356f0 msgr2=0x7fc8b013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8b01356f0 0x7fc8b013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient: _finish_hunting 0 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 monclient: found mon.noname-c 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 -- 172.21.15.138:0/2998367954 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b0140d20 con 0x7fc8b009e960 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b4fa6640 10 monclient: get_auth_request con 0x7fc8b00a32c0 auth_method 0 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b5fa8640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8b01356f0 0x7fc8b013ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b4fa6640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8b00a32c0 0x7fc8b013b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b009e960 msgr2=0x7fc8b013afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b57a7640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b009e960 0x7fc8b013afb0 secure :-1 s=READY pgs=5694 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac01ccf0 tx=0x7fc8ac019250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 client.?.objecter ms_handle_connect 0x7fc8b009e960 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0b0070 con 0x7fc8b009e960 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: dump: 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.257 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: _finish_auth 0 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8ac0b1040 con 0x7fc8b009e960 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0aa9e0 con 0x7fc8b009e960 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 5 monclient: authenticate success, global_id 22454 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a5ffb640 10 monclient: dump: 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.258 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient: _renew_subs 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8a77fe640 4 set_mon_vals no callback set 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b009e110 con 0x7fc8b009e960 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 client.22454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient: _renew_subs 2024-09-11T05:12:04.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b0141e00 con 0x7fc8b009e960 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 librados: init done 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0131e10 0x7fc8b01321f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.241+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b0073f10 con 0x7fc8b0131e10 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0131e10 0x7fc8b01321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 10 monclient: get_auth_request con 0x7fc8b0131e10 auth_method 0 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b57a7640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b009e960 0x7fc8b013afb0 secure :-1 s=READY pgs=5694 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac01ccf0 tx=0x7fc8ac019250 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7fc8ac0e9800 osd_map(205..205 src has 1..205) 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fc8ac0aac00 con 0x7fc8b009e960 2024-09-11T05:12:04.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_done global_id 22457 payload 824 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 1 -- 172.21.15.138:0/2998367954 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8b01414b0 con 0x7fc8b0131e10 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 1 --2- 172.21.15.138:0/2998367954 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc89403cef0 0x7fc89403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b5fa8640 1 --2- 172.21.15.138:0/2998367954 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc89403cef0 0x7fc89403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b5fa8640 10 monclient: get_auth_request con 0x7fc89403cef0 auth_method 0 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 10 client.22454.objecter ms_handle_connect 0x7fc8b0131e10 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 10 client.22454.objecter resend_mon_ops 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0131e10 msgr2=0x7fc8b01321f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b4fa6640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0131e10 0x7fc8b01321f0 secure :-1 s=READY pgs=5695 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00a1d90 tx=0x7fc8a0010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a000f040 con 0x7fc8b0131e10 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 10 monclient: dump: 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc8a001c400 con 0x7fc8b0131e10 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:04.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0131e10 msgr2=0x7fc8b01321f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0131e10 0x7fc8b01321f0 secure :-1 s=READY pgs=5695 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00a1d90 tx=0x7fc8a0010c20 comp rx=0 tx=0).stop 2024-09-11T05:12:04.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0131e10 msgr2=0x7fc8b01321f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:04.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0131e10 0x7fc8b01321f0 secure :-1 s=CLOSED pgs=5695 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00a1d90 tx=0x7fc8a0010c20 comp rx=0 tx=0).stop 2024-09-11T05:12:04.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8a5ffb640 10 client.22454.objecter ms_handle_connect 0x7fc89403cef0 2024-09-11T05:12:04.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.245+0000 7fc8b5fa8640 1 --2- 172.21.15.138:0/2998367954 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc89403cef0 0x7fc89403f3b0 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00a6400 tx=0x7fc8a8003d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b7a0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b7a0b640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0132730 0x7fc8b01db180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b7a0b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b009e420 con 0x7fc8b0132730 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0132730 0x7fc8b01db180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 10 monclient: get_auth_request con 0x7fc8b0132730 auth_method 0 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 10 monclient(hunting): handle_auth_done global_id 22460 payload 824 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 1 -- 172.21.15.138:0/2998367954 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc8b006cca0 con 0x7fc8b0132730 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0132730 msgr2=0x7fc8b01db180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8b4fa6640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0132730 0x7fc8b01db180 secure :-1 s=READY pgs=5696 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00a1d90 tx=0x7fc8a00046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8a5ffb640 10 client.22454.objecter ms_handle_connect 0x7fc8b0132730 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8a5ffb640 10 client.22454.objecter resend_mon_ops 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0005e60 con 0x7fc8b0132730 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.253+0000 7fc8a5ffb640 10 monclient: dump: 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.266 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fc8a0005e60 con 0x7fc8b0132730 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0132730 msgr2=0x7fc8b01db180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0132730 0x7fc8b01db180 secure :-1 s=READY pgs=5696 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00a1d90 tx=0x7fc8a00046d0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0132730 msgr2=0x7fc8b01db180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:04.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b0132730 0x7fc8b01db180 secure :-1 s=CLOSED pgs=5696 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00a1d90 tx=0x7fc8a00046d0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 20, 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 283025, 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.268 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.269 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.270 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:12:04.271 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.272 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.273 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.274 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:12:04.276 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:12:04.280 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:12:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 10 librados: watch_flush enter 2024-09-11T05:12:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 10 librados: watch_flush exit 2024-09-11T05:12:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 20 client.22454.objecter shutdown clearing up homeless session... 2024-09-11T05:12:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 10 client.22454.objecter successfully canceled tick 2024-09-11T05:12:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc89403cef0 msgr2=0x7fc89403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 --2- 172.21.15.138:0/2998367954 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc89403cef0 0x7fc89403f3b0 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00a6400 tx=0x7fc8a8003d50 comp rx=0 tx=0).stop 2024-09-11T05:12:04.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 10 monclient: shutdown 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b009e960 msgr2=0x7fc8b013afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 --2- 172.21.15.138:0/2998367954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8b009e960 0x7fc8b013afb0 secure :-1 s=READY pgs=5694 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac01ccf0 tx=0x7fc8ac019250 comp rx=0 tx=0).stop 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b5fa8640 1 -- 172.21.15.138:0/2998367954 reap_dead start 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 1 -- 172.21.15.138:0/2998367954 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(205..205 src has 1..205) ==== 5489+0+0 (unknown 0 0 0) 0x7fc8ac0e9800 con 0x7fc8b009e960 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 10 client.22454.objecter ms_dispatch 0x7fc8b0000ba0 osd_map(205..205 src has 1..205) 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8a5ffb640 10 librados: disconnected, discarding osd_map(205..205 src has 1..205) 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b5fa8640 1 -- 172.21.15.138:0/2998367954 reap_dead start 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 shutdown_connections 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 >> 172.21.15.138:0/2998367954 conn(0x7fc8b0063640 msgr2=0x7fc8b00a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 shutdown_connections 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 -- 172.21.15.138:0/2998367954 wait complete. 2024-09-11T05:12:04.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.257+0000 7fc8b7a0b640 1 librados: shutdown 2024-09-11T05:12:04.284 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:12:04.284 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:12:04.284 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:12:04.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.381+0000 7f641e55e640 10 monclient: get_monmap_and_config 2024-09-11T05:12:04.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.381+0000 7f641e55e640 10 monclient: build_initial_monmap 2024-09-11T05:12:04.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient: monmap: 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:04.385407+0000 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:04.385407+0000 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 1 Processor -- start 2024-09-11T05:12:04.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 1 -- start start 2024-09-11T05:12:04.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient: init 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6418070010 0x7f6418135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient(hunting): picked mon.noname-a con 0x7f6418070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f641809e6e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f641809eb10 0x7f64181389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient(hunting): picked mon.noname-c con 0x7f641809eb10 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64180a8990 con 0x7f641809e6e0 2024-09-11T05:12:04.388 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f64180703f0 con 0x7f6418070010 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f641809cbf0 con 0x7f641809eb10 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641e55e640 10 monclient(hunting): authenticate will time out at 1207.821220s 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f6417fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6418070010 0x7f6418135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f6417fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6418070010 0x7f6418135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35896/0 (socket says 172.21.15.138:35896) 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f6417fff640 1 -- 172.21.15.138:0/3145756936 learned_addr learned my addr 172.21.15.138:0/3145756936 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f6417fff640 10 monclient(hunting): get_auth_request con 0x7f6418070010 auth_method 0 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f6417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f6417fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f6417fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 1 --2- 172.21.15.138:0/3145756936 >> v2:172.21.15.190:3300/0 conn(0x7f641809eb10 0x7f64181389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient(hunting): get_auth_request con 0x7f641809eb10 auth_method 0 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient(hunting): handle_auth_done global_id 20584 payload 274 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 1 -- 172.21.15.138:0/3145756936 >> v2:172.21.15.138:3300/0 conn(0x7f6418070010 msgr2=0x7f6418135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 1 --2- 172.21.15.138:0/3145756936 >> v2:172.21.15.138:3300/0 conn(0x7f6418070010 0x7f6418135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 1 -- 172.21.15.138:0/3145756936 >> v1:172.21.15.138:6789/0 conn(0x7f641809e6e0 legacy=0x7f64181382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient: _finish_hunting 0 2024-09-11T05:12:04.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 1 monclient: found mon.noname-c 2024-09-11T05:12:04.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:04.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:12:04.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.385+0000 7f641cafb640 1 -- 172.21.15.138:0/3145756936 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641809ced0 con 0x7f641809eb10 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641cafb640 1 -- 172.21.15.138:0/3145756936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f641809eb10 msgr2=0x7f64181389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641cafb640 1 --2- 172.21.15.138:0/3145756936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f641809eb10 0x7f64181389d0 secure :-1 s=READY pgs=5779 cs=0 l=1 rev1=1 crypto rx=0x7f6418135200 tx=0x7f641001c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82ff79a2191960be server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 1 -- 172.21.15.138:0/3145756936 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64100b2070 con 0x7f641809eb10 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: dump: 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.393 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: _finish_auth 0 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: _check_auth_tickets 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 1 -- 172.21.15.138:0/3145756936 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64100a5860 con 0x7f641809eb10 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 5 monclient: authenticate success, global_id 20584 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 4 set_mon_vals no callback set 2024-09-11T05:12:04.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 1 -- 172.21.15.138:0/3145756936 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64100a5c30 con 0x7f641809eb10 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f64177fe640 10 monclient: dump: 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 10 monclient: shutdown 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 1 -- 172.21.15.138:0/3145756936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f641809eb10 msgr2=0x7f64181389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 1 --2- 172.21.15.138:0/3145756936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f641809eb10 0x7f64181389d0 secure :-1 s=READY pgs=5779 cs=0 l=1 rev1=1 crypto rx=0x7f6418135200 tx=0x7f641001c7c0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 1 -- 172.21.15.138:0/3145756936 shutdown_connections 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 1 --2- 172.21.15.138:0/3145756936 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f641809eb10 0x7f64181389d0 unknown :-1 s=CLOSED pgs=5779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 1 --2- 172.21.15.138:0/3145756936 >> v2:172.21.15.138:3300/0 conn(0x7f6418070010 0x7f6418135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 1 -- 172.21.15.138:0/3145756936 >> 172.21.15.138:0/3145756936 conn(0x7f64180ab830 msgr2=0x7f64180a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 1 -- 172.21.15.138:0/3145756936 shutdown_connections 2024-09-11T05:12:04.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.389+0000 7f641e55e640 1 -- 172.21.15.138:0/3145756936 wait complete. 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient: build_initial_monmap 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient: monmap: 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 librados: starting msgr at 2024-09-11T05:12:04.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 librados: starting objecter 2024-09-11T05:12:04.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 Processor -- start 2024-09-11T05:12:04.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 -- start start 2024-09-11T05:12:04.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 librados: setting wanted keys 2024-09-11T05:12:04.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 librados: calling monclient init 2024-09-11T05:12:04.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient: init 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6418070010 0x7f64180a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient(hunting): picked mon.noname-b con 0x7f6418070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f641809e6e0 0x7f64180a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient(hunting): picked mon.noname-c con 0x7f641809e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f641809eb10 0x7f64180a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient(hunting): picked mon.noname-a con 0x7f641809eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6418142e80 con 0x7f641809eb10 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6418139120 con 0x7f6418070010 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6418143000 con 0x7f641809e6e0 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641e55e640 10 monclient(hunting): authenticate will time out at 1207.829838s 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f641809eb10 0x7f64180a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f641809e6e0 0x7f64180a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f641809eb10 0x7f64180a4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35904/0 (socket says 172.21.15.138:35904) 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 -- 172.21.15.138:0/1277894535 learned_addr learned my addr 172.21.15.138:0/1277894535 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient(hunting): get_auth_request con 0x7f641809eb10 auth_method 0 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 10 monclient(hunting): get_auth_request con 0x7f641809e6e0 auth_method 0 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f6417fff640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6418070010 0x7f64180a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f6417fff640 10 monclient(hunting): get_auth_request con 0x7f6418070010 auth_method 0 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f6417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f6417fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f6417fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient(hunting): handle_auth_done global_id 64131 payload 824 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f641809e6e0 msgr2=0x7f64180a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f641809e6e0 0x7f64180a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6418070010 msgr2=0x7f64180a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6418070010 0x7f64180a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient: _finish_hunting 0 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 monclient: found mon.noname-a 2024-09-11T05:12:04.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f641cafb640 1 -- 172.21.15.138:0/1277894535 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641813b2b0 con 0x7f641809eb10 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.393+0000 7f640ffff640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f641809e6e0 0x7f64180a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641cafb640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f641809eb10 msgr2=0x7f64180a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641cafb640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f641809eb10 0x7f64180a4360 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64100b0040 tx=0x7f6410019560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 client.?.objecter ms_handle_connect 0x7f641809eb10 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64100b2050 con 0x7f641809eb10 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: dump: 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.407 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: _finish_auth 0 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: _check_auth_tickets 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64100be070 con 0x7f641809eb10 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:04.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64100adc60 con 0x7f641809eb10 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 5 monclient: authenticate success, global_id 64131 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 10 monclient: _renew_subs 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6418133000 con 0x7f641809eb10 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 monclient: dump: 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.409 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6416ffd640 4 set_mon_vals no callback set 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 10 client.64131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 10 monclient: _renew_subs 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f641813afd0 con 0x7f641809eb10 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 1 librados: init done 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d03e0 0x7f64181d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f64100a56e0 con 0x7f641809eb10 2024-09-11T05:12:04.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f641809d8d0 con 0x7f64181d03e0 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 1 --2- 172.21.15.138:0/1277894535 >> v2:172.21.15.190:6800/1562782189 conn(0x7f63fc03cef0 0x7f63fc03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5489+0+0 (secure 0 0 0) 0x7f64100ea570 con 0x7f641809eb10 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 10 client.64131.objecter ms_dispatch 0x7f6418000ba0 osd_map(205..205 src has 1..205) 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 3 client.64131.objecter handle_osd_map got epochs [205,205] > 0 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 3 client.64131.objecter handle_osd_map decoding full epoch 205 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f64157fa640 20 client.64131.objecter dump_active .. 0 homeless 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d03e0 0x7f64181d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 10 monclient: get_auth_request con 0x7f64181d03e0 auth_method 0 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f640ffff640 1 --2- 172.21.15.138:0/1277894535 >> v2:172.21.15.190:6800/1562782189 conn(0x7f63fc03cef0 0x7f63fc03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f640ffff640 10 monclient: get_auth_request con 0x7f63fc03cef0 auth_method 0 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 10 monclient(hunting): handle_auth_done global_id 20590 payload 824 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.397+0000 7f6417fff640 1 -- 172.21.15.138:0/1277894535 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6418135450 con 0x7f64181d03e0 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.401+0000 7f64157fa640 10 client.64131.objecter ms_handle_connect 0x7f64181d03e0 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.401+0000 7f64157fa640 10 client.64131.objecter resend_mon_ops 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.401+0000 7f6417fff640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d03e0 msgr2=0x7f64181d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.401+0000 7f6417fff640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d03e0 0x7f64181d07c0 secure :-1 s=READY pgs=5780 cs=0 l=1 rev1=1 crypto rx=0x7f640800f580 tx=0x7f640800fa40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.401+0000 7f64157fa640 10 client.64131.objecter ms_handle_connect 0x7f63fc03cef0 2024-09-11T05:12:04.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.401+0000 7f640ffff640 1 --2- 172.21.15.138:0/1277894535 >> v2:172.21.15.190:6800/1562782189 conn(0x7f63fc03cef0 0x7f63fc03f3b0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f64180a5720 tx=0x7f6400007e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.405+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408019030 con 0x7f64181d03e0 2024-09-11T05:12:04.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.405+0000 7f64157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.405+0000 7f64157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:12:04.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.405+0000 7f64157fa640 10 monclient: dump: 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.409+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6408004020 con 0x7f64181d03e0 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.409+0000 7f64157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.409+0000 7f64157fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.409+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d03e0 msgr2=0x7f64181d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.409+0000 7f64157fa640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d03e0 0x7f64181d07c0 secure :-1 s=READY pgs=5780 cs=0 l=1 rev1=1 crypto rx=0x7f640800f580 tx=0x7f640800fa40 comp rx=0 tx=0).stop 2024-09-11T05:12:04.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.409+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d03e0 msgr2=0x7f64181d07c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:04.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.409+0000 7f64157fa640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d03e0 0x7f64181d07c0 secure :-1 s=CLOSED pgs=5780 cs=0 l=1 rev1=1 crypto rx=0x7f640800f580 tx=0x7f640800fa40 comp rx=0 tx=0).stop 2024-09-11T05:12:04.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f641e55e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:12:04.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f641e55e640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d0d00 0x7f6418135e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f641e55e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64180a2330 con 0x7f64181d0d00 2024-09-11T05:12:04.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f6417fff640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d0d00 0x7f6418135e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f6417fff640 10 monclient: get_auth_request con 0x7f64181d0d00 auth_method 0 2024-09-11T05:12:04.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f6417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f6417fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.413+0000 7f6417fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f6417fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f6417fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f6417fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f6417fff640 10 monclient(hunting): handle_auth_done global_id 20593 payload 824 2024-09-11T05:12:04.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f6417fff640 1 -- 172.21.15.138:0/1277894535 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6418069fa0 con 0x7f64181d0d00 2024-09-11T05:12:04.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f6417fff640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d0d00 msgr2=0x7f6418135e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 10 client.64131.objecter ms_handle_connect 0x7f64181d0d00 2024-09-11T05:12:04.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 10 client.64131.objecter resend_mon_ops 2024-09-11T05:12:04.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f6417fff640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d0d00 0x7f6418135e90 secure :-1 s=READY pgs=5781 cs=0 l=1 rev1=1 crypto rx=0x7f640800f580 tx=0x7f64080046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640801fc20 con 0x7f64181d0d00 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 10 monclient: dump: 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.420 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f640801fc20 con 0x7f64181d0d00 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d0d00 msgr2=0x7f6418135e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d0d00 0x7f6418135e90 secure :-1 s=READY pgs=5781 cs=0 l=1 rev1=1 crypto rx=0x7f640800f580 tx=0x7f64080046d0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d0d00 msgr2=0x7f6418135e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.417+0000 7f64157fa640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64181d0d00 0x7f6418135e90 secure :-1 s=CLOSED pgs=5781 cs=0 l=1 rev1=1 crypto rx=0x7f640800f580 tx=0x7f64080046d0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.421 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 20, 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 283150, 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:12:04.422 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.423 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.424 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:12:04.425 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:04.426 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.427 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.428 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.430 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.431 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:04.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:12:04.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:12:04.434 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 10 librados: watch_flush enter 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 10 librados: watch_flush exit 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 20 client.64131.objecter shutdown clearing up homeless session... 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 10 client.64131.objecter successfully canceled tick 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 >> v2:172.21.15.190:6800/1562782189 conn(0x7f63fc03cef0 msgr2=0x7f63fc03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 --2- 172.21.15.138:0/1277894535 >> v2:172.21.15.190:6800/1562782189 conn(0x7f63fc03cef0 0x7f63fc03f3b0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f64180a5720 tx=0x7f6400007e40 comp rx=0 tx=0).stop 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 10 monclient: shutdown 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f641809eb10 msgr2=0x7f64180a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 --2- 172.21.15.138:0/1277894535 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f641809eb10 0x7f64180a4360 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64100b0040 tx=0x7f6410019560 comp rx=0 tx=0).stop 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641cafb640 1 -- 172.21.15.138:0/1277894535 reap_dead start 2024-09-11T05:12:04.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641cafb640 1 -- 172.21.15.138:0/1277894535 reap_dead start 2024-09-11T05:12:04.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 shutdown_connections 2024-09-11T05:12:04.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 >> 172.21.15.138:0/1277894535 conn(0x7f64180ab830 msgr2=0x7f64180a9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:04.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 shutdown_connections 2024-09-11T05:12:04.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 -- 172.21.15.138:0/1277894535 wait complete. 2024-09-11T05:12:04.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.421+0000 7f641e55e640 1 librados: shutdown 2024-09-11T05:12:04.437 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:12:04.437 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:12:04.437 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:12:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient: get_monmap_and_config 2024-09-11T05:12:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient: build_initial_monmap 2024-09-11T05:12:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient: monmap: 2024-09-11T05:12:04.546 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:04.547298+0000 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:04.547298+0000 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 1 Processor -- start 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 1 -- start start 2024-09-11T05:12:04.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient: init 2024-09-11T05:12:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:04.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c9c06e0e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6c9c09eb30 0x7f6c9c13a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9c09eb30 addr v2:172.21.15.138:3300/0 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c13ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9c09ef40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c9c09f540 con 0x7f6c9c06e0e0 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f6c9c06e4c0 con 0x7f6c9c09eb30 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9c09f820 con 0x7f6c9c09ef40 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6ca166c640 10 monclient(hunting): authenticate will time out at 1207.983271s 2024-09-11T05:12:04.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c13ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9a7fc640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6c9c09eb30 0x7f6c9c13a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c13ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44208/0 (socket says 172.21.15.138:44208) 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/1187105147 learned_addr learned my addr 172.21.15.138:0/1187105147 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient(hunting): get_auth_request con 0x7f6c9c09ef40 auth_method 0 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9a7fc640 10 monclient(hunting): get_auth_request con 0x7f6c9c09eb30 auth_method 0 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient(hunting): handle_auth_done global_id 22472 payload 274 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/1187105147 >> v2:172.21.15.138:3300/0 conn(0x7f6c9c09eb30 msgr2=0x7f6c9c13a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 1 --2- 172.21.15.138:0/1187105147 >> v2:172.21.15.138:3300/0 conn(0x7f6c9c09eb30 0x7f6c9c13a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/1187105147 >> v1:172.21.15.138:6789/0 conn(0x7f6c9c06e0e0 legacy=0x7f6c9c09e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 1 monclient: found mon.noname-b 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.545+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/1187105147 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c09fb00 con 0x7f6c9c09ef40 2024-09-11T05:12:04.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.549+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/1187105147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 msgr2=0x7f6c9c13ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.549+0000 7f6c9b7fe640 1 --2- 172.21.15.138:0/1187105147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c13ce80 secure :-1 s=READY pgs=5697 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0098a0 tx=0x7f6c8c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c2d1daefb3be29c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 1 -- 172.21.15.138:0/1187105147 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0b0070 con 0x7f6c9c09ef40 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: dump: 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: _finish_auth 0 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 1 -- 172.21.15.138:0/1187105147 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8c0ab600 con 0x7f6c9c09ef40 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 1 -- 172.21.15.138:0/1187105147 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0ab9d0 con 0x7f6c9c09ef40 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 5 monclient: authenticate success, global_id 22472 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 4 set_mon_vals no callback set 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c99ffb640 10 monclient: dump: 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.559 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient: shutdown 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- 172.21.15.138:0/1187105147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 msgr2=0x7f6c9c13ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 --2- 172.21.15.138:0/1187105147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c13ce80 secure :-1 s=READY pgs=5697 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0098a0 tx=0x7f6c8c01c400 comp rx=0 tx=0).stop 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- 172.21.15.138:0/1187105147 shutdown_connections 2024-09-11T05:12:04.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 --2- 172.21.15.138:0/1187105147 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c13ce80 unknown :-1 s=CLOSED pgs=5697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 --2- 172.21.15.138:0/1187105147 >> v2:172.21.15.138:3300/0 conn(0x7f6c9c09eb30 0x7f6c9c13a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- 172.21.15.138:0/1187105147 >> 172.21.15.138:0/1187105147 conn(0x7f6c9c063640 msgr2=0x7f6c9c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- 172.21.15.138:0/1187105147 shutdown_connections 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- 172.21.15.138:0/1187105147 wait complete. 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient: build_initial_monmap 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient: monmap: 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:04.561 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 librados: starting msgr at 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 librados: starting objecter 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 Processor -- start 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- start start 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 librados: setting wanted keys 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 librados: calling monclient init 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient: init 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c9c06e0e0 0x7f6c9c1d0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9c06e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:04.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c9c09eb30 0x7f6c9c133aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9c09eb30 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c06f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient(hunting): picked mon.noname-c con 0x7f6c9c09ef40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c13fd60 con 0x7f6c9c09eb30 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c13fbe0 con 0x7f6c9c06e0e0 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9c13fee0 con 0x7f6c9c09ef40 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6ca166c640 10 monclient(hunting): authenticate will time out at 1207.990229s 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c9c09eb30 0x7f6c9c133aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c06f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c9c09eb30 0x7f6c9c133aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35922/0 (socket says 172.21.15.138:35922) 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 1 -- 172.21.15.138:0/2744203616 learned_addr learned my addr 172.21.15.138:0/2744203616 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 10 monclient(hunting): get_auth_request con 0x7f6c9c09eb30 auth_method 0 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient(hunting): get_auth_request con 0x7f6c9c09ef40 auth_method 0 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9affd640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c9c06e0e0 0x7f6c9c1d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9affd640 10 monclient(hunting): get_auth_request con 0x7f6c9c06e0e0 auth_method 0 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient(hunting): handle_auth_done global_id 22475 payload 824 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/2744203616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c9c06e0e0 msgr2=0x7f6c9c1d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c9c06e0e0 0x7f6c9c1d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c9c09eb30 msgr2=0x7f6c9c133aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c9c09eb30 0x7f6c9c133aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 1 monclient: found mon.noname-c 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/2744203616 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c13c070 con 0x7f6c9c09ef40 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.553+0000 7f6c9a7fc640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c9c09eb30 0x7f6c9c133aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c9b7fe640 1 -- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 msgr2=0x7f6c9c06f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c9b7fe640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c06f400 secure :-1 s=READY pgs=5698 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0a6e80 tx=0x7f6c8c0191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 client.?.objecter ms_handle_connect 0x7f6c9c09ef40 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 1 -- 172.21.15.138:0/2744203616 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0b0070 con 0x7f6c9c09ef40 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:04.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: dump: 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: _finish_auth 0 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: _check_auth_tickets 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:04.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 5 monclient: authenticate success, global_id 22475 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 1 -- 172.21.15.138:0/2744203616 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8c0b1040 con 0x7f6c9c09ef40 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 10 monclient: _renew_subs 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9c13c350 con 0x7f6c9c09ef40 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 1 -- 172.21.15.138:0/2744203616 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0aaa00 con 0x7f6c9c09ef40 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 monclient: dump: 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:04.568 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c98ff9640 4 set_mon_vals no callback set 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 10 client.22475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 10 monclient: _renew_subs 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9c134300 con 0x7f6c9c09ef40 2024-09-11T05:12:04.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 1 librados: init done 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 1 -- 172.21.15.138:0/2744203616 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f6c8c0aaba0 con 0x7f6c9c09ef40 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c9c13c7e0 con 0x7f6c9c09ef40 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 1 --2- 172.21.15.138:0/2744203616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c7c03cf20 0x7f6c7c03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 1 -- 172.21.15.138:0/2744203616 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(206..206 src has 1..206) ==== 5699+0+0 (secure 0 0 0) 0x7f6c8c0e9a50 con 0x7f6c9c09ef40 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 10 client.22475.objecter ms_dispatch 0x7f6c9c000ba0 osd_map(206..206 src has 1..206) 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 3 client.22475.objecter handle_osd_map got epochs [206,206] > 0 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 3 client.22475.objecter handle_osd_map decoding full epoch 206 2024-09-11T05:12:04.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c937fe640 20 client.22475.objecter dump_active .. 0 homeless 2024-09-11T05:12:04.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c9affd640 1 --2- 172.21.15.138:0/2744203616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c7c03cf20 0x7f6c7c03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:04.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.557+0000 7f6c9affd640 10 monclient: get_auth_request con 0x7f6c7c03cf20 auth_method 0 2024-09-11T05:12:04.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.561+0000 7f6c937fe640 10 client.22475.objecter ms_handle_connect 0x7f6c7c03cf20 2024-09-11T05:12:04.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.561+0000 7f6c9affd640 1 --2- 172.21.15.138:0/2744203616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c7c03cf20 0x7f6c7c03f3e0 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f6c94009f20 tx=0x7f6c940094d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:04.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.561+0000 7f6c937fe640 1 -- 172.21.15.138:0/2744203616 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f6c8c016110 con 0x7f6c9c09ef40 2024-09-11T05:12:04.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.561+0000 7f6c937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:04.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.561+0000 7f6c937fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:04.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.785+0000 7f6ca166c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:12:04.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.785+0000 7f6ca166c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:12:04.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.785+0000 7f6ca166c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:04.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.785+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6c9c13a6b0 con 0x7f6c9c09ef40 2024-09-11T05:12:04.791 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.791 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=312012143,config=692619927,health=3203186241,kv=2712226084,logm=2694555506}) 2024-09-11T05:12:04.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f6c937fe640 1 -- 172.21.15.138:0/2744203616 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6c8c0b5df0 con 0x7f6c9c09ef40 2024-09-11T05:12:04.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f6c937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:12:04.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f6c937fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:04.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=1,health=11,kv=9,logm=66} crc {auth=312012143,config=1228700967,health=2201294010,kv=2863542489,logm=3348631479}) 2024-09-11T05:12:04.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=312012143,config=692619927,health=3203186241,kv=2712226084,logm=2694555506}) 2024-09-11T05:12:04.793 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.789+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=2595665215,config=692619927,health=1440552615,kv=310889196,logm=1774639247}) 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 10 librados: watch_flush enter 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 10 librados: watch_flush exit 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 20 client.22475.objecter shutdown clearing up homeless session... 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 10 client.22475.objecter successfully canceled tick 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c7c03cf20 msgr2=0x7f6c7c03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 --2- 172.21.15.138:0/2744203616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c7c03cf20 0x7f6c7c03f3e0 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f6c94009f20 tx=0x7f6c940094d0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 10 monclient: shutdown 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 msgr2=0x7f6c9c06f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:04.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c06f400 secure :-1 s=READY pgs=5698 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0a6e80 tx=0x7f6c8c0191a0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 shutdown_connections 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 --2- 172.21.15.138:0/2744203616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c7c03cf20 0x7f6c7c03f3e0 unknown :-1 s=CLOSED pgs=3364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6c9c09ef40 0x7f6c9c06f400 unknown :-1 s=CLOSED pgs=5698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6c9c09eb30 0x7f6c9c133aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 --2- 172.21.15.138:0/2744203616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6c9c06e0e0 0x7f6c9c1d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 >> 172.21.15.138:0/2744203616 conn(0x7f6c9c063640 msgr2=0x7f6c9c0a21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 shutdown_connections 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 -- 172.21.15.138:0/2744203616 wait complete. 2024-09-11T05:12:04.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f6ca166c640 1 librados: shutdown 2024-09-11T05:12:04.796 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.796 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748950141}) 2024-09-11T05:12:04.796 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3645357048}) 2024-09-11T05:12:04.796 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.796 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748950141}) 2024-09-11T05:12:04.796 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.793+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630810318}) 2024-09-11T05:12:04.798 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.797+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.798 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.797+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511350614}) 2024-09-11T05:12:04.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.797+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381525663}) 2024-09-11T05:12:04.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.797+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.797+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511350614}) 2024-09-11T05:12:04.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.797+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324724435}) 2024-09-11T05:12:04.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.801+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.801+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922774896}) 2024-09-11T05:12:04.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.801+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2028614442}) 2024-09-11T05:12:04.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.801+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.801+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922774896}) 2024-09-11T05:12:04.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.801+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2863710278}) 2024-09-11T05:12:04.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040391231}) 2024-09-11T05:12:04.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1498745706}) 2024-09-11T05:12:04.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040391231}) 2024-09-11T05:12:04.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404167355}) 2024-09-11T05:12:04.807 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.807 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945513602}) 2024-09-11T05:12:04.807 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005109999}) 2024-09-11T05:12:04.807 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.807 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945513602}) 2024-09-11T05:12:04.807 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.805+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275384042}) 2024-09-11T05:12:04.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:12:04.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.809+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.809+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274929737}) 2024-09-11T05:12:04.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.809+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469388051}) 2024-09-11T05:12:04.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.809+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.809+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274929737}) 2024-09-11T05:12:04.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.809+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287080770}) 2024-09-11T05:12:04.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860818817}) 2024-09-11T05:12:04.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070463983}) 2024-09-11T05:12:04.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860818817}) 2024-09-11T05:12:04.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613131529}) 2024-09-11T05:12:04.816 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.816 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015535562}) 2024-09-11T05:12:04.816 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4200442688}) 2024-09-11T05:12:04.816 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.816 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015535562}) 2024-09-11T05:12:04.816 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.813+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1025350023}) 2024-09-11T05:12:04.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.817+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.817+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714244834}) 2024-09-11T05:12:04.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.817+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155157827}) 2024-09-11T05:12:04.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.817+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.817+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714244834}) 2024-09-11T05:12:04.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.817+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2048527854}) 2024-09-11T05:12:04.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.821+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.821+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905919854}) 2024-09-11T05:12:04.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.821+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731989554}) 2024-09-11T05:12:04.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.821+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.821+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905919854}) 2024-09-11T05:12:04.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.821+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1735923900}) 2024-09-11T05:12:04.825 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.825 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440003583}) 2024-09-11T05:12:04.825 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2041322138}) 2024-09-11T05:12:04.825 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.825 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440003583}) 2024-09-11T05:12:04.825 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1919465485}) 2024-09-11T05:12:04.827 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.827 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428315175}) 2024-09-11T05:12:04.827 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=916996995}) 2024-09-11T05:12:04.828 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.828 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428315175}) 2024-09-11T05:12:04.828 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.825+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202464262}) 2024-09-11T05:12:04.830 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.829+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.830 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.829+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448923362}) 2024-09-11T05:12:04.831 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.829+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1049836606}) 2024-09-11T05:12:04.831 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.829+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.831 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.829+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448923362}) 2024-09-11T05:12:04.831 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.829+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778986769}) 2024-09-11T05:12:04.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.833+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.833+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106261624}) 2024-09-11T05:12:04.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.833+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2339022400}) 2024-09-11T05:12:04.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.833+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.833+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106261624}) 2024-09-11T05:12:04.834 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.833+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347737078}) 2024-09-11T05:12:04.836 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.833+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.836 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.833+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466687833}) 2024-09-11T05:12:04.836 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1503633817}) 2024-09-11T05:12:04.836 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.836 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466687833}) 2024-09-11T05:12:04.836 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=902634296}) 2024-09-11T05:12:04.838 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.838 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299361345}) 2024-09-11T05:12:04.838 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766535453}) 2024-09-11T05:12:04.838 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.838 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299361345}) 2024-09-11T05:12:04.838 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.837+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739872214}) 2024-09-11T05:12:04.841 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.841 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745294686}) 2024-09-11T05:12:04.841 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404632992}) 2024-09-11T05:12:04.841 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.841 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745294686}) 2024-09-11T05:12:04.841 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728007361}) 2024-09-11T05:12:04.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869708168}) 2024-09-11T05:12:04.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3191278566}) 2024-09-11T05:12:04.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869708168}) 2024-09-11T05:12:04.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300073936}) 2024-09-11T05:12:04.846 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.845+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.846 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.845+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709504766}) 2024-09-11T05:12:04.846 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.845+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68499155}) 2024-09-11T05:12:04.846 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.845+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.846 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.845+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709504766}) 2024-09-11T05:12:04.846 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.845+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903650962}) 2024-09-11T05:12:04.848 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.849 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086108245}) 2024-09-11T05:12:04.849 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4251093629}) 2024-09-11T05:12:04.849 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.849 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086108245}) 2024-09-11T05:12:04.849 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1519558539}) 2024-09-11T05:12:04.851 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.851 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3166727608}) 2024-09-11T05:12:04.851 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3719972935}) 2024-09-11T05:12:04.851 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.851 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3166727608}) 2024-09-11T05:12:04.851 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.849+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1885024692}) 2024-09-11T05:12:04.853 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.853+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.854 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.853+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252528583}) 2024-09-11T05:12:04.854 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.853+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3081189848}) 2024-09-11T05:12:04.854 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.853+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.854 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.853+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252528583}) 2024-09-11T05:12:04.854 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.853+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2183185465}) 2024-09-11T05:12:04.857 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.857+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.857 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.857+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=27} crc {logm=60649646,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2210664746,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3668423285,osd_snap=2980751559,osdmap=3875732188}) 2024-09-11T05:12:04.857 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.857+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=31} crc {logm=60649646,mdsmap=1754867906,mgr=1796167710,mgr_metadata=568375954,mgrstat=3803129563,mon_config_key=1174248985,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3668423285,osd_snap=1913478061,osdmap=2953491445}) 2024-09-11T05:12:04.857 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.857+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.857 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.857+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=27} crc {logm=60649646,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2210664746,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3668423285,osd_snap=2980751559,osdmap=3875732188}) 2024-09-11T05:12:04.857 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.857+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=29} crc {logm=60649646,mdsmap=1754867906,mgr=2081689912,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=485203287,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3668423285,osd_snap=184419471,osdmap=2316408219}) 2024-09-11T05:12:04.860 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.860 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=544407776}) 2024-09-11T05:12:04.860 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=40887505}) 2024-09-11T05:12:04.860 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.860 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=544407776}) 2024-09-11T05:12:04.860 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2586229100}) 2024-09-11T05:12:04.865 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.865 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2565521720}) 2024-09-11T05:12:04.865 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=407034595}) 2024-09-11T05:12:04.865 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.865 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2565521720}) 2024-09-11T05:12:04.866 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.861+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1250974121}) 2024-09-11T05:12:04.877 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.877+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.877 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.877+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=1503055004}) 2024-09-11T05:12:04.877 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.877+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=91} crc {osdmap=3147813301}) 2024-09-11T05:12:04.877 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.877+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:12:04.877 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.877+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=91} crc {osdmap=1503055004}) 2024-09-11T05:12:04.878 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:12:04.877+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=88} crc {osdmap=1787789498}) 2024-09-11T05:12:05.810 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:12:05.811 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient: get_monmap_and_config 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient: build_initial_monmap 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient: monmap: 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:05.942162+0000 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:05.942162+0000 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:05.941 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:05.942 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:05.942 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:05.942 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:05.942 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:05.942 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:05.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 1 Processor -- start 2024-09-11T05:12:05.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 1 -- start start 2024-09-11T05:12:05.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient: init 2024-09-11T05:12:05.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb09806e0e0 0x7fb098136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient(hunting): picked mon.noname-c con 0x7fb09806e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb098136170 0x7fb098136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient(hunting): picked mon.noname-b con 0x7fb098136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0980a1920 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0980a3270 con 0x7fb0980a1920 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb09806e4c0 con 0x7fb09806e0e0 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb0980a0af0 con 0x7fb098136170 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09e443640 10 monclient(hunting): authenticate will time out at 1209.379004s 2024-09-11T05:12:05.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb098136170 0x7fb098136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb098136170 0x7fb098136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44232/0 (socket says 172.21.15.138:44232) 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb09c9e0640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fb0980a1920 0x7fb0980a1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33620/0 (socket says 172.21.15.138:33620) 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 1 -- 172.21.15.138:0/370027822 learned_addr learned my addr 172.21.15.138:0/370027822 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient(hunting): get_auth_request con 0x7fb098136170 auth_method 0 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb097fff640 1 --2- 172.21.15.138:0/370027822 >> v2:172.21.15.190:3300/0 conn(0x7fb09806e0e0 0x7fb098136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb097fff640 10 monclient(hunting): get_auth_request con 0x7fb09806e0e0 auth_method 0 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb097fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:05.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb097fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb097fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient(hunting): handle_auth_done global_id 22478 payload 274 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 1 -- 172.21.15.138:0/370027822 >> v2:172.21.15.190:3300/0 conn(0x7fb09806e0e0 msgr2=0x7fb098136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 1 --2- 172.21.15.138:0/370027822 >> v2:172.21.15.190:3300/0 conn(0x7fb09806e0e0 0x7fb098136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 1 -- 172.21.15.138:0/370027822 >> v1:172.21.15.138:6789/0 conn(0x7fb0980a1920 legacy=0x7fb0980a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient: _finish_hunting 0 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 1 monclient: found mon.noname-b 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:05.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.941+0000 7fb0977fe640 1 -- 172.21.15.138:0/370027822 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0980a2f90 con 0x7fb098136170 2024-09-11T05:12:05.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb0977fe640 1 -- 172.21.15.138:0/370027822 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb098136170 msgr2=0x7fb098136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:05.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb0977fe640 1 --2- 172.21.15.138:0/370027822 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb098136170 0x7fb098136c60 secure :-1 s=READY pgs=5701 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009870 tx=0x7fb08c01c240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9dc03294f3b3aa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:05.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 1 -- 172.21.15.138:0/370027822 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c0b0070 con 0x7fb098136170 2024-09-11T05:12:05.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:05.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:05.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: dump: 2024-09-11T05:12:05.951 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:05.952 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: _finish_auth 0 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 1 -- 172.21.15.138:0/370027822 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb08c0ab600 con 0x7fb098136170 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb09e443640 5 monclient: authenticate success, global_id 22478 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb09e443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 1 -- 172.21.15.138:0/370027822 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c0ab9d0 con 0x7fb098136170 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb09e443640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:05.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb09e443640 4 set_mon_vals no callback set 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.945+0000 7fb096ffd640 10 monclient: dump: 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient: shutdown 2024-09-11T05:12:05.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- 172.21.15.138:0/370027822 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb098136170 msgr2=0x7fb098136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 --2- 172.21.15.138:0/370027822 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb098136170 0x7fb098136c60 secure :-1 s=READY pgs=5701 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009870 tx=0x7fb08c01c240 comp rx=0 tx=0).stop 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- 172.21.15.138:0/370027822 shutdown_connections 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 --2- 172.21.15.138:0/370027822 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb098136170 0x7fb098136c60 unknown :-1 s=CLOSED pgs=5701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 --2- 172.21.15.138:0/370027822 >> v2:172.21.15.190:3300/0 conn(0x7fb09806e0e0 0x7fb098136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- 172.21.15.138:0/370027822 >> 172.21.15.138:0/370027822 conn(0x7fb098063640 msgr2=0x7fb098063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- 172.21.15.138:0/370027822 shutdown_connections 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- 172.21.15.138:0/370027822 wait complete. 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient: build_initial_monmap 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient: monmap: 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:05.955 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 librados: starting msgr at 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 librados: starting objecter 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 Processor -- start 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- start start 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 librados: setting wanted keys 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 librados: calling monclient init 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient: init 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:05.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb09806e0e0 0x7fb09813d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient(hunting): picked mon.noname-c con 0x7fb09806e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0980a1920 0x7fb0981d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient(hunting): picked mon.noname-a con 0x7fb0980a1920 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb098136170 0x7fb0981d1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient(hunting): picked mon.noname-b con 0x7fb098136170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0981410a0 con 0x7fb0980a1920 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb098140f20 con 0x7fb098136170 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb098141220 con 0x7fb09806e0e0 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:05.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09e443640 10 monclient(hunting): authenticate will time out at 1209.386004s 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb09806e0e0 0x7fb09813d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0980a1920 0x7fb0981d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb09806e0e0 0x7fb09813d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44234/0 (socket says 172.21.15.138:44234) 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 -- 172.21.15.138:0/3364264662 learned_addr learned my addr 172.21.15.138:0/3364264662 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient(hunting): get_auth_request con 0x7fb09806e0e0 auth_method 0 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 10 monclient(hunting): get_auth_request con 0x7fb0980a1920 auth_method 0 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:05.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09c9e0640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb098136170 0x7fb0981d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09c9e0640 10 monclient(hunting): get_auth_request con 0x7fb098136170 auth_method 0 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09c9e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09c9e0640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb09c9e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:05.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient(hunting): handle_auth_done global_id 22481 payload 824 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 -- 172.21.15.138:0/3364264662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb098136170 msgr2=0x7fb0981d1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb098136170 0x7fb0981d1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 -- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0980a1920 msgr2=0x7fb0981d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0980a1920 0x7fb0981d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient: _finish_hunting 0 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 monclient: found mon.noname-c 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb097fff640 1 -- 172.21.15.138:0/3364264662 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09813ca70 con 0x7fb09806e0e0 2024-09-11T05:12:05.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.949+0000 7fb0977fe640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0980a1920 0x7fb0981d0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb097fff640 1 -- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb09806e0e0 msgr2=0x7fb09813d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb097fff640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb09806e0e0 0x7fb09813d180 secure :-1 s=READY pgs=5702 cs=0 l=1 rev1=1 crypto rx=0x7fb08800d5f0 tx=0x7fb08800dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 client.?.objecter ms_handle_connect 0x7fb09806e0e0 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 1 -- 172.21.15.138:0/3364264662 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb088012800 con 0x7fb09806e0e0 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: dump: 2024-09-11T05:12:05.963 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: _finish_auth 0 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:12:05.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 5 monclient: authenticate success, global_id 22481 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 1 -- 172.21.15.138:0/3364264662 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb088019070 con 0x7fb09806e0e0 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 1 -- 172.21.15.138:0/3364264662 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb088012be0 con 0x7fb09806e0e0 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb0967fc640 4 set_mon_vals no callback set 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 monclient: dump: 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:05.965 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 10 monclient: _renew_subs 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09809e110 con 0x7fb09806e0e0 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 10 client.22481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 10 monclient: _renew_subs 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb09813a710 con 0x7fb09806e0e0 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 1 librados: init done 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:05.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0980a1db0 con 0x7fb09806e0e0 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 1 -- 172.21.15.138:0/3364264662 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fb088013070 con 0x7fb09806e0e0 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 1 --2- 172.21.15.138:0/3364264662 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb08403cf20 0x7fb08403f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 1 -- 172.21.15.138:0/3364264662 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(207..207 src has 1..207) ==== 5789+0+0 (secure 0 0 0) 0x7fb08804fc20 con 0x7fb09806e0e0 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 10 client.22481.objecter ms_dispatch 0x7fb098000ba0 osd_map(207..207 src has 1..207) 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 3 client.22481.objecter handle_osd_map got epochs [207,207] > 0 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 3 client.22481.objecter handle_osd_map decoding full epoch 207 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb094ff9640 20 client.22481.objecter dump_active .. 0 homeless 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb0977fe640 1 --2- 172.21.15.138:0/3364264662 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb08403cf20 0x7fb08403f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.953+0000 7fb0977fe640 10 monclient: get_auth_request con 0x7fb08403cf20 auth_method 0 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.957+0000 7fb0977fe640 1 --2- 172.21.15.138:0/3364264662 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb08403cf20 0x7fb08403f3e0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7fb098130830 tx=0x7fb08c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.957+0000 7fb094ff9640 10 client.22481.objecter ms_handle_connect 0x7fb08403cf20 2024-09-11T05:12:05.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.957+0000 7fb094ff9640 1 -- 172.21.15.138:0/3364264662 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fb088012d80 con 0x7fb09806e0e0 2024-09-11T05:12:05.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.957+0000 7fb094ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:05.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:05.957+0000 7fb094ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:06.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:12:06.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:06.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:06.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb09806ccd0 con 0x7fb09806e0e0 2024-09-11T05:12:06.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb094ff9640 1 -- 172.21.15.138:0/3364264662 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fb08801ec10 con 0x7fb09806e0e0 2024-09-11T05:12:06.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb094ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:06.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb094ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:06.198 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:12:06.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 10 librados: watch_flush enter 2024-09-11T05:12:06.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 10 librados: watch_flush exit 2024-09-11T05:12:06.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 20 client.22481.objecter shutdown clearing up homeless session... 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 10 client.22481.objecter successfully canceled tick 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb08403cf20 msgr2=0x7fb08403f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 --2- 172.21.15.138:0/3364264662 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb08403cf20 0x7fb08403f3e0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7fb098130830 tx=0x7fb08c008040 comp rx=0 tx=0).stop 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 10 monclient: shutdown 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb09806e0e0 msgr2=0x7fb09813d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb09806e0e0 0x7fb09813d180 secure :-1 s=READY pgs=5702 cs=0 l=1 rev1=1 crypto rx=0x7fb08800d5f0 tx=0x7fb08800dab0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 shutdown_connections 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 --2- 172.21.15.138:0/3364264662 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb08403cf20 0x7fb08403f3e0 unknown :-1 s=CLOSED pgs=3365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb098136170 0x7fb0981d1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0980a1920 0x7fb0981d0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 --2- 172.21.15.138:0/3364264662 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb09806e0e0 0x7fb09813d180 unknown :-1 s=CLOSED pgs=5702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.197+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 >> 172.21.15.138:0/3364264662 conn(0x7fb098063640 msgr2=0x7fb09813da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:06.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.201+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 shutdown_connections 2024-09-11T05:12:06.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.201+0000 7fb09e443640 1 -- 172.21.15.138:0/3364264662 wait complete. 2024-09-11T05:12:06.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.201+0000 7fb09e443640 1 librados: shutdown 2024-09-11T05:12:06.216 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:12:06.216 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:12:06.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:12:06.216 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:12:06.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient: get_monmap_and_config 2024-09-11T05:12:06.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient: build_initial_monmap 2024-09-11T05:12:06.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient: monmap: 2024-09-11T05:12:06.333 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:06.333 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.333 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:06.334152+0000 2024-09-11T05:12:06.333 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:06.334152+0000 2024-09-11T05:12:06.333 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:06.333 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.334 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:06.334 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:06.334 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:06.334 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:06.334 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:06.334 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 1 Processor -- start 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 1 -- start start 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient: init 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9560070010 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f956009e6e0 0x7f956013d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient(hunting): picked mon.noname-a con 0x7f956009e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f956009eb10 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:06.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9560135b60 con 0x7f9560070010 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95600a8990 con 0x7f956009eb10 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f95600703f0 con 0x7f956009e6e0 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9567576640 10 monclient(hunting): authenticate will time out at 1209.770188s 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f956009e6e0 0x7f956013d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f956009e6e0 0x7f956013d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35950/0 (socket says 172.21.15.138:35950) 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 1 -- 172.21.15.138:0/1841927789 learned_addr learned my addr 172.21.15.138:0/1841927789 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 10 monclient(hunting): get_auth_request con 0x7f956009e6e0 auth_method 0 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.333+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9565312640 1 -- 172.21.15.138:0/1841927789 >> v1:172.21.15.138:6789/0 conn(0x7f9560070010 legacy=0x7f9560135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9565b13640 1 -- 172.21.15.138:0/1841927789 >> v1:172.21.15.190:6789/0 conn(0x7f956009eb10 legacy=0x7f956013fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9564b11640 10 monclient(hunting): handle_auth_done global_id 64146 payload 274 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9564b11640 1 -- 172.21.15.138:0/1841927789 >> v1:172.21.15.190:6789/0 conn(0x7f956009eb10 legacy=0x7f956013fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9564b11640 1 -- 172.21.15.138:0/1841927789 >> v1:172.21.15.138:6789/0 conn(0x7f9560070010 legacy=0x7f9560135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9564b11640 10 monclient: _finish_hunting 0 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9564b11640 1 monclient: found mon.noname-a 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9564b11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9564b11640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:06.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.337+0000 7f9564b11640 1 -- 172.21.15.138:0/1841927789 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956009cbf0 con 0x7f956009e6e0 2024-09-11T05:12:06.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9564b11640 1 -- 172.21.15.138:0/1841927789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f956009e6e0 msgr2=0x7f956013d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:06.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9564b11640 1 --2- 172.21.15.138:0/1841927789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f956009e6e0 0x7f956013d640 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f955c009970 tx=0x7f955c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fee2aa97ad8787 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:06.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 1 -- 172.21.15.138:0/1841927789 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c0aa730 con 0x7f956009e6e0 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: dump: 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: _finish_auth 0 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: _check_auth_tickets 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 1 -- 172.21.15.138:0/1841927789 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f955c0af070 con 0x7f956009e6e0 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 1 -- 172.21.15.138:0/1841927789 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c0aaaf0 con 0x7f956009e6e0 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 5 monclient: authenticate success, global_id 64146 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 4 set_mon_vals no callback set 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9553fff640 10 monclient: dump: 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.343 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.344 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.344 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.344 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.344 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.344 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.344 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.344 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient: shutdown 2024-09-11T05:12:06.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:06.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- 172.21.15.138:0/1841927789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f956009e6e0 msgr2=0x7f956013d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 --2- 172.21.15.138:0/1841927789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f956009e6e0 0x7f956013d640 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f955c009970 tx=0x7f955c01c4f0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- 172.21.15.138:0/1841927789 shutdown_connections 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 --2- 172.21.15.138:0/1841927789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f956009e6e0 0x7f956013d640 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- 172.21.15.138:0/1841927789 >> 172.21.15.138:0/1841927789 conn(0x7f95600ab830 msgr2=0x7f95600a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- 172.21.15.138:0/1841927789 shutdown_connections 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- 172.21.15.138:0/1841927789 wait complete. 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient: build_initial_monmap 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient: monmap: 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:06.352 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 librados: starting msgr at 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 librados: starting objecter 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 Processor -- start 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- start start 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 librados: setting wanted keys 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 librados: calling monclient init 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient: init 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560070010 0x7f95601320e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient(hunting): picked mon.noname-a con 0x7f9560070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:06.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f956009e6e0 0x7f9560136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient(hunting): picked mon.noname-b con 0x7f956009e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:06.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f956009eb10 0x7f95601372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient(hunting): picked mon.noname-c con 0x7f956009eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:06.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9560143e60 con 0x7f9560070010 2024-09-11T05:12:06.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9560143ce0 con 0x7f956009e6e0 2024-09-11T05:12:06.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f9560143fe0 con 0x7f956009eb10 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9567576640 10 monclient(hunting): authenticate will time out at 1209.779842s 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9565312640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560070010 0x7f95601320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9565b13640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f956009eb10 0x7f95601372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.341+0000 7f9565312640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560070010 0x7f95601320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35958/0 (socket says 172.21.15.138:35958) 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 -- 172.21.15.138:0/2625583822 learned_addr learned my addr 172.21.15.138:0/2625583822 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient(hunting): get_auth_request con 0x7f9560070010 auth_method 0 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565b13640 10 monclient(hunting): get_auth_request con 0x7f956009eb10 auth_method 0 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565b13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565b13640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565b13640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565b13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565b13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565b13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f956009e6e0 0x7f9560136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): get_auth_request con 0x7f956009e6e0 auth_method 0 2024-09-11T05:12:06.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient(hunting): handle_auth_done global_id 64149 payload 824 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f956009eb10 msgr2=0x7f95601372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f956009eb10 0x7f95601372a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f956009e6e0 msgr2=0x7f9560136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f956009e6e0 0x7f9560136d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient: _finish_hunting 0 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 monclient: found mon.noname-a 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 -- 172.21.15.138:0/2625583822 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95600aaac0 con 0x7f9560070010 2024-09-11T05:12:06.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560070010 msgr2=0x7f95601320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560070010 0x7f95601320e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9554001ec0 tx=0x7f955400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 client.?.objecter ms_handle_connect 0x7f9560070010 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9554017070 con 0x7f9560070010 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: dump: 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.358 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: _finish_auth 0 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 5 monclient: authenticate success, global_id 64149 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 10 monclient: _renew_subs 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9560138820 con 0x7f9560070010 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95540057d0 con 0x7f9560070010 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:06.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9554005b80 con 0x7f9560070010 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 10 monclient: dump: 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.360 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9552ffd640 4 set_mon_vals no callback set 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 10 client.64149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 10 monclient: _renew_subs 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9560139380 con 0x7f9560070010 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 1 librados: init done 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:06.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560132620 0x7f95601377e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f956013ac90 con 0x7f9560132620 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9565312640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560070010 0x7f95601320e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9554001ec0 tx=0x7f955400c960 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f955401b3b0 mgrmap(e 5) 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560132620 0x7f95601377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient: get_auth_request con 0x7f9560132620 auth_method 0 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f955401b3b0 con 0x7f9560070010 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.345+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9564b11640 10 monclient(hunting): handle_auth_done global_id 64152 payload 824 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9564b11640 1 -- 172.21.15.138:0/2625583822 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9560135450 con 0x7f9560132620 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 1 --2- 172.21.15.138:0/2625583822 >> v2:172.21.15.190:6800/1562782189 conn(0x7f954403cea0 0x7f954403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9565b13640 1 --2- 172.21.15.138:0/2625583822 >> v2:172.21.15.190:6800/1562782189 conn(0x7f954403cea0 0x7f954403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9565b13640 10 monclient: get_auth_request con 0x7f954403cea0 auth_method 0 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9564b11640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560132620 msgr2=0x7f95601377e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 10 client.64149.objecter ms_handle_connect 0x7f9560132620 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 10 client.64149.objecter resend_mon_ops 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9564b11640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560132620 0x7f95601377e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f955c0ae000 tx=0x7f955c019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:06.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c0b9070 con 0x7f9560132620 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 10 monclient: dump: 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.364 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f955c0af040 con 0x7f9560132620 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560132620 msgr2=0x7f95601377e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560132620 0x7f95601377e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f955c0ae000 tx=0x7f955c019290 comp rx=0 tx=0).stop 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560132620 msgr2=0x7f95601377e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:06.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560132620 0x7f95601377e0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f955c0ae000 tx=0x7f955c019290 comp rx=0 tx=0).stop 2024-09-11T05:12:06.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9551ffb640 10 client.64149.objecter ms_handle_connect 0x7f954403cea0 2024-09-11T05:12:06.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.349+0000 7f9565b13640 1 --2- 172.21.15.138:0/2625583822 >> v2:172.21.15.190:6800/1562782189 conn(0x7f954403cea0 0x7f954403f360 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f95580012f0 tx=0x7f9558009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:06.367 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9567576640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9567576640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560137d20 0x7f956009f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9567576640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9560138100 con 0x7f9560137d20 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560137d20 0x7f956009f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 10 monclient: get_auth_request con 0x7f9560137d20 auth_method 0 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 10 monclient(hunting): handle_auth_done global_id 64155 payload 824 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 1 -- 172.21.15.138:0/2625583822 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9560069fa0 con 0x7f9560137d20 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560137d20 msgr2=0x7f956009f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9564b11640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560137d20 0x7f956009f0b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f955c009970 tx=0x7f955c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9551ffb640 10 client.64149.objecter ms_handle_connect 0x7f9560137d20 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.357+0000 7f9551ffb640 10 client.64149.objecter resend_mon_ops 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c0c4a80 con 0x7f9560137d20 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 10 monclient: dump: 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f955c0a6040 con 0x7f9560137d20 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560137d20 msgr2=0x7f956009f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560137d20 0x7f956009f0b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f955c009970 tx=0x7f955c009240 comp rx=0 tx=0).stop 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560137d20 msgr2=0x7f956009f0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:06.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560137d20 0x7f956009f0b0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f955c009970 tx=0x7f955c009240 comp rx=0 tx=0).stop 2024-09-11T05:12:06.371 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:12:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:12:06.371 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 20, 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 29987, 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 4, 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:06.372 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:06.373 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:06.374 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:06.375 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:12:06.376 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:06.377 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:06.378 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:06.379 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:06.380 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:06.381 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:06.382 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:12:06.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:06.384 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:12:06.385 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:12:06.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 10 librados: watch_flush enter 2024-09-11T05:12:06.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 10 librados: watch_flush exit 2024-09-11T05:12:06.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 20 client.64149.objecter shutdown clearing up homeless session... 2024-09-11T05:12:06.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 10 client.64149.objecter successfully canceled tick 2024-09-11T05:12:06.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 >> v2:172.21.15.190:6800/1562782189 conn(0x7f954403cea0 msgr2=0x7f954403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 --2- 172.21.15.138:0/2625583822 >> v2:172.21.15.190:6800/1562782189 conn(0x7f954403cea0 0x7f954403f360 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f95580012f0 tx=0x7f9558009040 comp rx=0 tx=0).stop 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 10 monclient: shutdown 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560070010 msgr2=0x7f95601320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 --2- 172.21.15.138:0/2625583822 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9560070010 0x7f95601320e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9554001ec0 tx=0x7f955400c960 comp rx=0 tx=0).stop 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9565b13640 1 -- 172.21.15.138:0/2625583822 reap_dead start 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 1 -- 172.21.15.138:0/2625583822 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 5789+0+0 (unknown 0 0 0) 0x7f955404f0b0 con 0x7f9560070010 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 10 client.64149.objecter ms_dispatch 0x7f9560000ba0 osd_map(207..207 src has 1..207) 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9551ffb640 10 librados: disconnected, discarding osd_map(207..207 src has 1..207) 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9565b13640 1 -- 172.21.15.138:0/2625583822 reap_dead start 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 shutdown_connections 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 >> 172.21.15.138:0/2625583822 conn(0x7f95600ab830 msgr2=0x7f95600a9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 shutdown_connections 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 -- 172.21.15.138:0/2625583822 wait complete. 2024-09-11T05:12:06.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.361+0000 7f9567576640 1 librados: shutdown 2024-09-11T05:12:06.388 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:12:06.388 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:12:06.388 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:12:06.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.513+0000 7f258dbf0640 10 monclient: get_monmap_and_config 2024-09-11T05:12:06.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.513+0000 7f258dbf0640 10 monclient: build_initial_monmap 2024-09-11T05:12:06.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.513+0000 7f258dbf0640 10 monclient: monmap: 2024-09-11T05:12:06.516 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:06.516 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.516 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:06.517320+0000 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:06.517320+0000 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:06.517 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 1 Processor -- start 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 1 -- start start 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient: init 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25880a32c0 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f258813a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient(hunting): picked mon.noname-b con 0x7f258809eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f258809ef50 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25880a1b50 con 0x7f258809ef50 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f258809f560 con 0x7f25880a32c0 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f25880a1920 con 0x7f258809eb30 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f258dbf0640 10 monclient(hunting): authenticate will time out at 1209.953286s 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2587fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f258809ef50 0x7f258813ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33638/0 (socket says 172.21.15.138:33638) 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 1 --2- 172.21.15.138:0/3058222181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f258813a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2587fff640 1 -- 172.21.15.138:0/3058222181 learned_addr learned my addr 172.21.15.138:0/3058222181 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient(hunting): get_auth_request con 0x7f258809eb30 auth_method 0 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient(hunting): handle_auth_done global_id 22487 payload 274 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 1 -- 172.21.15.138:0/3058222181 >> v1:172.21.15.190:6789/0 conn(0x7f25880a32c0 legacy=0x7f258809e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 1 -- 172.21.15.138:0/3058222181 >> v1:172.21.15.138:6789/0 conn(0x7f258809ef50 legacy=0x7f258813ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 1 monclient: found mon.noname-b 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.517+0000 7f2586ffd640 1 -- 172.21.15.138:0/3058222181 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258809f840 con 0x7f258809eb30 2024-09-11T05:12:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f2586ffd640 1 -- 172.21.15.138:0/3058222181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 msgr2=0x7f258813a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f2586ffd640 1 --2- 172.21.15.138:0/3058222181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f258813a930 secure :-1 s=READY pgs=5703 cs=0 l=1 rev1=1 crypto rx=0x7f257c0097f0 tx=0x7f257c01c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9063b31241769a80 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 1 -- 172.21.15.138:0/3058222181 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c0b0070 con 0x7f258809eb30 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: dump: 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: _finish_auth 0 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: _check_auth_tickets 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 1 -- 172.21.15.138:0/3058222181 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f257c0ab600 con 0x7f258809eb30 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 1 -- 172.21.15.138:0/3058222181 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c0ab9d0 con 0x7f258809eb30 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 5 monclient: authenticate success, global_id 22487 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 4 set_mon_vals no callback set 2024-09-11T05:12:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f25867fc640 10 monclient: dump: 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 10 monclient: shutdown 2024-09-11T05:12:06.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 1 -- 172.21.15.138:0/3058222181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 msgr2=0x7f258813a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 1 --2- 172.21.15.138:0/3058222181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f258813a930 secure :-1 s=READY pgs=5703 cs=0 l=1 rev1=1 crypto rx=0x7f257c0097f0 tx=0x7f257c01c430 comp rx=0 tx=0).stop 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 1 -- 172.21.15.138:0/3058222181 shutdown_connections 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 1 --2- 172.21.15.138:0/3058222181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f258813a930 unknown :-1 s=CLOSED pgs=5703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 1 -- 172.21.15.138:0/3058222181 >> 172.21.15.138:0/3058222181 conn(0x7f2588063640 msgr2=0x7f2588063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 1 -- 172.21.15.138:0/3058222181 shutdown_connections 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 1 -- 172.21.15.138:0/3058222181 wait complete. 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 10 monclient: build_initial_monmap 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.521+0000 7f258dbf0640 10 monclient: monmap: 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:06.527 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 librados: starting msgr at 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 librados: starting objecter 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 Processor -- start 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 -- start start 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 librados: setting wanted keys 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 librados: calling monclient init 2024-09-11T05:12:06.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient: init 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f2588135f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient(hunting): picked mon.noname-c con 0x7f258809eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f258809ef50 0x7f25881414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient(hunting): picked mon.noname-b con 0x7f258809ef50 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25880a32c0 0x7f2588141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient(hunting): picked mon.noname-a con 0x7f25880a32c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2588140f40 con 0x7f25880a32c0 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2588140dc0 con 0x7f258809ef50 2024-09-11T05:12:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f25881410c0 con 0x7f258809eb30 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient(hunting): authenticate will time out at 1209.960816s 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f2588135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2587fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25880a32c0 0x7f2588141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f2588135f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44252/0 (socket says 172.21.15.138:44252) 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 -- 172.21.15.138:0/2853612932 learned_addr learned my addr 172.21.15.138:0/2853612932 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient(hunting): get_auth_request con 0x7f258809eb30 auth_method 0 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2587fff640 10 monclient(hunting): get_auth_request con 0x7f25880a32c0 auth_method 0 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2587fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient(hunting): handle_auth_done global_id 22490 payload 824 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f258809ef50 0x7f25881414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f258809ef50 msgr2=0x7f25881414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f258809ef50 0x7f25881414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25880a32c0 msgr2=0x7f2588141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f25880a32c0 0x7f2588141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient: _finish_hunting 0 2024-09-11T05:12:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 monclient: found mon.noname-c 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 -- 172.21.15.138:0/2853612932 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258813d000 con 0x7f258809eb30 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2586ffd640 10 monclient: get_auth_request con 0x7f258809ef50 auth_method 0 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f258809ef50 0x7f25881414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 msgr2=0x7f2588135f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f25877fe640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f2588135f90 secure :-1 s=READY pgs=5704 cs=0 l=1 rev1=1 crypto rx=0x7f257800c450 tx=0x7f257800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 client.?.objecter ms_handle_connect 0x7f258809eb30 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2578012a40 con 0x7f258809eb30 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: dump: 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: _finish_auth 0 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:12:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2578017070 con 0x7f258809eb30 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 5 monclient: authenticate success, global_id 22490 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient: _renew_subs 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f258813bc20 con 0x7f258809eb30 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2578012e40 con 0x7f258809eb30 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f256ffff640 10 monclient: dump: 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f2585ffb640 4 set_mon_vals no callback set 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 client.22490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient: _renew_subs 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f258809e110 con 0x7f258809eb30 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.525+0000 7f258dbf0640 1 librados: init done 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f258dbf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f258dbf0640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f258dbf0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2588142340 con 0x7f2588141f60 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 10 monclient: get_auth_request con 0x7f2588141f60 auth_method 0 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f257801c460 con 0x7f258809eb30 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 10 monclient(hunting): handle_auth_done global_id 22493 payload 824 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 1 -- 172.21.15.138:0/2853612932 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f258809e4a0 con 0x7f2588141f60 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 msgr2=0x7f2588136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f257c0ad000 tx=0x7f257c0191d0 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f256ffff640 1 --2- 172.21.15.138:0/2853612932 >> v2:172.21.15.190:6800/1562782189 conn(0x7f257403ced0 0x7f257403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(208..208 src has 1..208) ==== 6583+0+0 (secure 0 0 0) 0x7f257804fbe0 con 0x7f258809eb30 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f256ffff640 10 client.22490.objecter ms_dispatch 0x7f2588000ba0 osd_map(208..208 src has 1..208) 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f256ffff640 3 client.22490.objecter handle_osd_map got epochs [208,208] > 0 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f256ffff640 3 client.22490.objecter handle_osd_map decoding full epoch 208 2024-09-11T05:12:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f256ffff640 20 client.22490.objecter dump_active .. 0 homeless 2024-09-11T05:12:06.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2587fff640 1 --2- 172.21.15.138:0/2853612932 >> v2:172.21.15.190:6800/1562782189 conn(0x7f257403ced0 0x7f257403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.529+0000 7f2587fff640 10 monclient: get_auth_request con 0x7f257403ced0 auth_method 0 2024-09-11T05:12:06.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.533+0000 7f2587fff640 1 --2- 172.21.15.138:0/2853612932 >> v2:172.21.15.190:6800/1562782189 conn(0x7f257403ced0 0x7f257403f390 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f2580009400 tx=0x7f2580009350 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:06.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.533+0000 7f256ffff640 10 client.22490.objecter ms_handle_connect 0x7f257403ced0 2024-09-11T05:12:06.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:06.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 10 monclient: get_auth_request con 0x7f2588141f60 auth_method 0 2024-09-11T05:12:06.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:06.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 10 monclient: _init_auth method 2 2024-09-11T05:12:06.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 10 monclient: _init_auth already have auth, reseting 2024-09-11T05:12:06.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 10 monclient: handle_auth_reply_more payload 9 2024-09-11T05:12:06.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-11T05:12:06.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-11T05:12:06.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 10 monclient: handle_auth_done global_id 22493 payload 293 2024-09-11T05:12:06.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 msgr2=0x7f2588136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:12:06.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 msgr2=0x7f2588136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:12:06.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f257c000c10 tx=0x7f257c008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:12:06.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:06.733+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:12:07.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 10 monclient: get_auth_request con 0x7f2588141f60 auth_method 0 2024-09-11T05:12:07.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 10 monclient: _init_auth method 2 2024-09-11T05:12:07.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 10 monclient: _init_auth already have auth, reseting 2024-09-11T05:12:07.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 10 monclient: handle_auth_reply_more payload 9 2024-09-11T05:12:07.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-11T05:12:07.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.137+0000 7f2586ffd640 10 monclient: handle_auth_done global_id 22493 payload 293 2024-09-11T05:12:07.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f2586ffd640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 msgr2=0x7f2588136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 secure :-1 s=READY pgs=5705 cs=0 l=1 rev1=1 crypto rx=0x7f257c01c260 tx=0x7f257c0b4000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 10 client.22490.objecter ms_handle_connect 0x7f2588141f60 2024-09-11T05:12:07.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 10 client.22490.objecter resend_mon_ops 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c0b0040 con 0x7f2588141f60 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 10 monclient: dump: 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.141 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.142 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f257c0c0620 con 0x7f2588141f60 2024-09-11T05:12:07.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:07.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:07.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 msgr2=0x7f2588136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 secure :-1 s=READY pgs=5705 cs=0 l=1 rev1=1 crypto rx=0x7f257c01c260 tx=0x7f257c0b4000 comp rx=0 tx=0).stop 2024-09-11T05:12:07.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 msgr2=0x7f2588136910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:07.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.141+0000 7f256ffff640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588141f60 0x7f2588136910 secure :-1 s=CLOSED pgs=5705 cs=0 l=1 rev1=1 crypto rx=0x7f257c01c260 tx=0x7f257c0b4000 comp rx=0 tx=0).stop 2024-09-11T05:12:07.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f258dbf0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:12:07.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f258dbf0640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588136e50 0x7f2588137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f258dbf0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2588142e90 con 0x7f2588136e50 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588136e50 0x7f2588137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 10 monclient: get_auth_request con 0x7f2588136e50 auth_method 0 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:07.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 10 monclient(hunting): handle_auth_done global_id 22496 payload 824 2024-09-11T05:12:07.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 1 -- 172.21.15.138:0/2853612932 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f25881364d0 con 0x7f2588136e50 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588136e50 msgr2=0x7f2588137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f2586ffd640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588136e50 0x7f2588137230 secure :-1 s=READY pgs=5706 cs=0 l=1 rev1=1 crypto rx=0x7f257c0097f0 tx=0x7f258809ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f256ffff640 10 client.22490.objecter ms_handle_connect 0x7f2588136e50 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.149+0000 7f256ffff640 10 client.22490.objecter resend_mon_ops 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c0c5040 con 0x7f2588136e50 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 10 monclient: dump: 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.152 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.153 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.153 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.153 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.153 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.153 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.153 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f257c0c6810 con 0x7f2588136e50 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588136e50 msgr2=0x7f2588137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588136e50 0x7f2588137230 secure :-1 s=READY pgs=5706 cs=0 l=1 rev1=1 crypto rx=0x7f257c0097f0 tx=0x7f258809ff80 comp rx=0 tx=0).stop 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588136e50 msgr2=0x7f2588137230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f256ffff640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2588136e50 0x7f2588137230 secure :-1 s=CLOSED pgs=5706 cs=0 l=1 rev1=1 crypto rx=0x7f257c0097f0 tx=0x7f258809ff80 comp rx=0 tx=0).stop 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 20, 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 285925, 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 5, 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:12:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:12:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:12:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:12:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:12:07.166 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:12:07.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 10 librados: watch_flush enter 2024-09-11T05:12:07.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 10 librados: watch_flush exit 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 20 client.22490.objecter shutdown clearing up homeless session... 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 10 client.22490.objecter successfully canceled tick 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 >> v2:172.21.15.190:6800/1562782189 conn(0x7f257403ced0 msgr2=0x7f257403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 --2- 172.21.15.138:0/2853612932 >> v2:172.21.15.190:6800/1562782189 conn(0x7f257403ced0 0x7f257403f390 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f2580009400 tx=0x7f2580009350 comp rx=0 tx=0).stop 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 10 monclient: shutdown 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 msgr2=0x7f2588135f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 --2- 172.21.15.138:0/2853612932 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f258809eb30 0x7f2588135f90 secure :-1 s=READY pgs=5704 cs=0 l=1 rev1=1 crypto rx=0x7f257800c450 tx=0x7f257800c910 comp rx=0 tx=0).stop 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f2587fff640 1 -- 172.21.15.138:0/2853612932 reap_dead start 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f2587fff640 1 -- 172.21.15.138:0/2853612932 reap_dead start 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 shutdown_connections 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 >> 172.21.15.138:0/2853612932 conn(0x7f2588063640 msgr2=0x7f25880a5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:07.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 shutdown_connections 2024-09-11T05:12:07.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 -- 172.21.15.138:0/2853612932 wait complete. 2024-09-11T05:12:07.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.153+0000 7f258dbf0640 1 librados: shutdown 2024-09-11T05:12:07.176 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:12:07.176 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:12:07.176 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:12:07.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient: get_monmap_and_config 2024-09-11T05:12:07.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient: build_initial_monmap 2024-09-11T05:12:07.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient: monmap: 2024-09-11T05:12:07.301 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:07.301 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.301 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:07.302612+0000 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:07.302612+0000 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:07.302 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 1 Processor -- start 2024-09-11T05:12:07.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 1 -- start start 2024-09-11T05:12:07.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient: init 2024-09-11T05:12:07.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809da20 0x7fd6880a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient(hunting): picked mon.noname-b con 0x7fd68809da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6880a1e60 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6880a22a0 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd688064590 con 0x7fd6880a1e60 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6880a2c70 con 0x7fd6880a22a0 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd68809de00 con 0x7fd68809da20 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd6901f0640 10 monclient(hunting): authenticate will time out at 1210.738742s 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68d78b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fd6880a1e60 0x7fd688136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33654/0 (socket says 172.21.15.138:33654) 2024-09-11T05:12:07.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809da20 0x7fd6880a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68d78b640 1 -- 172.21.15.138:0/2044815375 learned_addr learned my addr 172.21.15.138:0/2044815375 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient(hunting): get_auth_request con 0x7fd68809da20 auth_method 0 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient(hunting): handle_auth_done global_id 22499 payload 274 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 1 -- 172.21.15.138:0/2044815375 >> v1:172.21.15.190:6789/0 conn(0x7fd6880a22a0 legacy=0x7fd688136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 1 -- 172.21.15.138:0/2044815375 >> v1:172.21.15.138:6789/0 conn(0x7fd6880a1e60 legacy=0x7fd688136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient: _finish_hunting 0 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 1 monclient: found mon.noname-b 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:07.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:07.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.300+0000 7fd68df8c640 1 -- 172.21.15.138:0/2044815375 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6880a2f50 con 0x7fd68809da20 2024-09-11T05:12:07.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 -- 172.21.15.138:0/2044815375 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809da20 msgr2=0x7fd6880a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 --2- 172.21.15.138:0/2044815375 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809da20 0x7fd6880a1920 secure :-1 s=READY pgs=5707 cs=0 l=1 rev1=1 crypto rx=0x7fd680009dc0 tx=0x7fd68001c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16e0d1e7a891918 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 1 -- 172.21.15.138:0/2044815375 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800b2070 con 0x7fd68809da20 2024-09-11T05:12:07.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:07.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: dump: 2024-09-11T05:12:07.308 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: _finish_auth 0 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: _check_auth_tickets 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 1 -- 172.21.15.138:0/2044815375 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6800a58a0 con 0x7fd68809da20 2024-09-11T05:12:07.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 1 -- 172.21.15.138:0/2044815375 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800a5c70 con 0x7fd68809da20 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 5 monclient: authenticate success, global_id 22499 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 4 set_mon_vals no callback set 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68cf8a640 10 monclient: dump: 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.310 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.311 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.311 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient: shutdown 2024-09-11T05:12:07.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:07.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- 172.21.15.138:0/2044815375 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809da20 msgr2=0x7fd6880a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 --2- 172.21.15.138:0/2044815375 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809da20 0x7fd6880a1920 secure :-1 s=READY pgs=5707 cs=0 l=1 rev1=1 crypto rx=0x7fd680009dc0 tx=0x7fd68001c820 comp rx=0 tx=0).stop 2024-09-11T05:12:07.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- 172.21.15.138:0/2044815375 shutdown_connections 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 --2- 172.21.15.138:0/2044815375 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809da20 0x7fd6880a1920 unknown :-1 s=CLOSED pgs=5707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- 172.21.15.138:0/2044815375 >> 172.21.15.138:0/2044815375 conn(0x7fd688063640 msgr2=0x7fd688063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- 172.21.15.138:0/2044815375 shutdown_connections 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- 172.21.15.138:0/2044815375 wait complete. 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient: build_initial_monmap 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient: monmap: 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.317 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 librados: starting msgr at 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 librados: starting objecter 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 Processor -- start 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- start start 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 librados: setting wanted keys 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 librados: calling monclient init 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient: init 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6880a1e60 0x7fd68813ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient(hunting): picked mon.noname-a con 0x7fd6880a1e60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6880a22a0 0x7fd68809e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient(hunting): picked mon.noname-b con 0x7fd6880a22a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809e960 0x7fd68809edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient(hunting): picked mon.noname-c con 0x7fd68809e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd688141d70 con 0x7fd6880a1e60 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd688141ef0 con 0x7fd6880a22a0 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd688142070 con 0x7fd68809e960 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd6901f0640 10 monclient(hunting): authenticate will time out at 1210.746695s 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6880a1e60 0x7fd68813ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809e960 0x7fd68809edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6880a1e60 0x7fd68813ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35996/0 (socket says 172.21.15.138:35996) 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809e960 0x7fd68809edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44294/0 (socket says 172.21.15.138:44294) 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 -- 172.21.15.138:0/3747406318 learned_addr learned my addr 172.21.15.138:0/3747406318 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient(hunting): get_auth_request con 0x7fd6880a1e60 auth_method 0 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 10 monclient(hunting): get_auth_request con 0x7fd68809e960 auth_method 0 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68d78b640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6880a22a0 0x7fd68809e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68d78b640 10 monclient(hunting): get_auth_request con 0x7fd6880a22a0 auth_method 0 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68d78b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68d78b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68d78b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient(hunting): handle_auth_done global_id 64161 payload 824 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809e960 msgr2=0x7fd68809edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809e960 0x7fd68809edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6880a22a0 msgr2=0x7fd68809e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6880a22a0 0x7fd68809e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient: _finish_hunting 0 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 monclient: found mon.noname-a 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68df8c640 1 -- 172.21.15.138:0/3747406318 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68809e110 con 0x7fd6880a1e60 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.308+0000 7fd68e78d640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd68809e960 0x7fd68809edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68df8c640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6880a1e60 msgr2=0x7fd68813ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68df8c640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6880a1e60 0x7fd68813ee90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6800b0040 tx=0x7fd6800195c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 client.?.objecter ms_handle_connect 0x7fd6880a1e60 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800bd020 con 0x7fd6880a1e60 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: dump: 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.322 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: _finish_auth 0 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: _check_auth_tickets 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 5 monclient: authenticate success, global_id 64161 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6800b8880 con 0x7fd6880a1e60 2024-09-11T05:12:07.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 10 monclient: _renew_subs 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd688071ce0 con 0x7fd6880a1e60 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800b2040 con 0x7fd6880a1e60 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 monclient: dump: 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.324 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 10 client.64161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 10 monclient: _renew_subs 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd688070ca0 con 0x7fd6880a1e60 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6877fe640 4 set_mon_vals no callback set 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 1 librados: init done 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd68812d9a0 0x7fd68812dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd688137160 con 0x7fd68812d9a0 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fd6800b8c70 con 0x7fd6880a1e60 2024-09-11T05:12:07.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd68812d9a0 0x7fd68812dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 10 monclient: get_auth_request con 0x7fd68812d9a0 auth_method 0 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 1 --2- 172.21.15.138:0/3747406318 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd67403cea0 0x7fd67403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 6583+0+0 (secure 0 0 0) 0x7fd6800eb100 con 0x7fd6880a1e60 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 10 client.64161.objecter ms_dispatch 0x7fd688000ba0 osd_map(208..208 src has 1..208) 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 3 client.64161.objecter handle_osd_map got epochs [208,208] > 0 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 3 client.64161.objecter handle_osd_map decoding full epoch 208 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd6867fc640 20 client.64161.objecter dump_active .. 0 homeless 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68e78d640 1 --2- 172.21.15.138:0/3747406318 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd67403cea0 0x7fd67403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68e78d640 10 monclient: get_auth_request con 0x7fd67403cea0 auth_method 0 2024-09-11T05:12:07.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 10 monclient(hunting): handle_auth_done global_id 20620 payload 824 2024-09-11T05:12:07.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.312+0000 7fd68d78b640 1 -- 172.21.15.138:0/3747406318 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd688074390 con 0x7fd68812d9a0 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd68e78d640 1 --2- 172.21.15.138:0/3747406318 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd67403cea0 0x7fd67403f360 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7fd678007fd0 tx=0x7fd678007dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 10 client.64161.objecter ms_handle_connect 0x7fd67403cea0 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 10 client.64161.objecter ms_handle_connect 0x7fd68812d9a0 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 10 client.64161.objecter resend_mon_ops 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd68d78b640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd68812d9a0 msgr2=0x7fd68812dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd68d78b640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd68812d9a0 0x7fd68812dd80 secure :-1 s=READY pgs=5787 cs=0 l=1 rev1=1 crypto rx=0x7fd688134c60 tx=0x7fd67c00e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c004e40 con 0x7fd68812d9a0 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 10 monclient: dump: 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.331 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd67c018070 con 0x7fd68812d9a0 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd68812d9a0 msgr2=0x7fd68812dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd68812d9a0 0x7fd68812dd80 secure :-1 s=READY pgs=5787 cs=0 l=1 rev1=1 crypto rx=0x7fd688134c60 tx=0x7fd67c00e960 comp rx=0 tx=0).stop 2024-09-11T05:12:07.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd68812d9a0 msgr2=0x7fd68812dd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.316+0000 7fd6867fc640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd68812d9a0 0x7fd68812dd80 secure :-1 s=CLOSED pgs=5787 cs=0 l=1 rev1=1 crypto rx=0x7fd688134c60 tx=0x7fd67c00e960 comp rx=0 tx=0).stop 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd6901f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd6901f0640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688072190 0x7fd68812e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd6901f0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd68809fcd0 con 0x7fd688072190 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688072190 0x7fd68812e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 10 monclient: get_auth_request con 0x7fd688072190 auth_method 0 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 10 monclient(hunting): handle_auth_done global_id 20623 payload 824 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.324+0000 7fd68d78b640 1 -- 172.21.15.138:0/3747406318 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6880a19a0 con 0x7fd688072190 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 10 client.64161.objecter ms_handle_connect 0x7fd688072190 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 10 client.64161.objecter resend_mon_ops 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd68d78b640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688072190 msgr2=0x7fd68812e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd68d78b640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688072190 0x7fd68812e2c0 secure :-1 s=READY pgs=5788 cs=0 l=1 rev1=1 crypto rx=0x7fd68812c730 tx=0x7fd67c004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0066f0 con 0x7fd688072190 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 10 monclient: dump: 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.334 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fd67c0066f0 con 0x7fd688072190 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688072190 msgr2=0x7fd68812e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688072190 0x7fd68812e2c0 secure :-1 s=READY pgs=5788 cs=0 l=1 rev1=1 crypto rx=0x7fd68812c730 tx=0x7fd67c004950 comp rx=0 tx=0).stop 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688072190 msgr2=0x7fd68812e2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:07.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6867fc640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd688072190 0x7fd68812e2c0 secure :-1 s=CLOSED pgs=5788 cs=0 l=1 rev1=1 crypto rx=0x7fd68812c730 tx=0x7fd67c004950 comp rx=0 tx=0).stop 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 20, 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 286062, 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:12:07.336 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 5, 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:07.337 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:07.338 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:12:07.339 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:07.340 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.341 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.342 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.343 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:07.344 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.347 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:07.348 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:07.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:07.349 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:12:07.349 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:12:07.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 10 librados: watch_flush enter 2024-09-11T05:12:07.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 10 librados: watch_flush exit 2024-09-11T05:12:07.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 20 client.64161.objecter shutdown clearing up homeless session... 2024-09-11T05:12:07.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 10 client.64161.objecter successfully canceled tick 2024-09-11T05:12:07.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd67403cea0 msgr2=0x7fd67403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 --2- 172.21.15.138:0/3747406318 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd67403cea0 0x7fd67403f360 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7fd678007fd0 tx=0x7fd678007dc0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 10 monclient: shutdown 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6880a1e60 msgr2=0x7fd68813ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6880a1e60 0x7fd68813ee90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6800b0040 tx=0x7fd6800195c0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd68e78d640 1 -- 172.21.15.138:0/3747406318 reap_dead start 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 shutdown_connections 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 --2- 172.21.15.138:0/3747406318 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd6880a1e60 0x7fd68813ee90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 >> 172.21.15.138:0/3747406318 conn(0x7fd688063640 msgr2=0x7fd6880a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 shutdown_connections 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 -- 172.21.15.138:0/3747406318 wait complete. 2024-09-11T05:12:07.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.328+0000 7fd6901f0640 1 librados: shutdown 2024-09-11T05:12:07.351 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:12:07.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T05:12:07.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T05:12:07.352 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T05:12:07.352 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T05:12:07.352 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T05:12:07.352 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:12:07.379 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T05:12:07.379 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:12:07.380 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:12:07.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient: get_monmap_and_config 2024-09-11T05:12:07.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient: build_initial_monmap 2024-09-11T05:12:07.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient: monmap: 2024-09-11T05:12:07.493 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:07.493 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.493 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:07.494487+0000 2024-09-11T05:12:07.493 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:07.494487+0000 2024-09-11T05:12:07.493 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 1 Processor -- start 2024-09-11T05:12:07.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 1 -- start start 2024-09-11T05:12:07.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient: init 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7effe80a0a90 0x7effe80a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient(hunting): picked mon.noname-c con 0x7effe80a0a90 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7effe809dc20 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7effe8139be0 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe809efa0 con 0x7effe809dc20 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe80a2d80 con 0x7effe8139be0 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7effe809ecc0 con 0x7effe80a0a90 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:07.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effedea8640 10 monclient(hunting): authenticate will time out at 1210.931385s 2024-09-11T05:12:07.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effdffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7effe809dc20 0x7effe80a2670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33656/0 (socket says 172.21.15.138:33656) 2024-09-11T05:12:07.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effe77fe640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7effe80a0a90 msgr2=0x7effe80a0e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:07.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effdffff640 1 -- 172.21.15.138:0/2533770176 learned_addr learned my addr 172.21.15.138:0/2533770176 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:12:07.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effe77fe640 1 --2- 172.21.15.138:0/2533770176 >> v2:172.21.15.190:3300/0 conn(0x7effe80a0a90 0x7effe80a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:07.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.492+0000 7effe7fff640 1 -- 172.21.15.138:0/2533770176 >> v1:172.21.15.190:6789/0 conn(0x7effe8139be0 legacy=0x7effe813bfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:12:07.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effdffff640 1 -- 172.21.15.138:0/2533770176 >> v1:172.21.15.138:6789/0 conn(0x7effe809dc20 legacy=0x7effe80a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968286458 0 0) 0x7effd0001460 con 0x7effe809dc20 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): _negotiate 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): my global_id is 64164 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effd4003530 con 0x7effe809dc20 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1971097858 0 0) 0x7effd4003530 con 0x7effe809dc20 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 >> v2:172.21.15.190:3300/0 conn(0x7effe80a0a90 msgr2=0x7effe80a0e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 --2- 172.21.15.138:0/2533770176 >> v2:172.21.15.190:3300/0 conn(0x7effe80a0a90 0x7effe80a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 >> v1:172.21.15.190:6789/0 conn(0x7effe8139be0 legacy=0x7effe813bfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:12:07.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe80a13b0 con 0x7effe809dc20 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: _finish_auth 0 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7effd0001c40 con 0x7effe809dc20 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effedea8640 5 monclient: authenticate success, global_id 64164 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effedea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: dump: 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.502 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 >> v1:172.21.15.138:6789/0 conn(0x7effe809dc20 legacy=0x7effe80a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effd40039c0 0x7effd4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): picked mon.a con 0x7effd40039c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd4004810 0x7effd4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): picked mon.b con 0x7effd4004810 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effd4005380 0x7effd400b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): picked mon.c con 0x7effd4005380 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7effe809ecc0 con 0x7effd40039c0 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7effd400c2d0 con 0x7effd4004810 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7effd400c4d0 con 0x7effd4005380 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:12:07.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effedea8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effd40039c0 0x7effd4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient(hunting): get_auth_request con 0x7effd40039c0 auth_method 0 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effdffff640 1 -- 172.21.15.138:0/2533770176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd4004810 msgr2=0x7effd4004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effd4005380 0x7effd400b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effdffff640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd4004810 0x7effd4004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 10 monclient(hunting): get_auth_request con 0x7effd4005380 auth_method 0 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:12:07.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient(hunting): handle_auth_done global_id 64164 payload 293 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 1 -- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effd4005380 msgr2=0x7effd400b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effd4005380 0x7effd400b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 1 -- 172.21.15.138:0/2533770176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd4004810 msgr2=0x7effd4004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd4004810 0x7effd4004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient: _finish_hunting 0 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 1 monclient: found mon.a 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:07.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:07.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 1 -- 172.21.15.138:0/2533770176 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd4003530 con 0x7effd40039c0 2024-09-11T05:12:07.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient: _finish_auth 0 2024-09-11T05:12:07.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 10 monclient: _check_auth_tickets 2024-09-11T05:12:07.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:07.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.496+0000 7effe7fff640 1 -- 172.21.15.138:0/2533770176 reap_dead start 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe77fe640 1 -- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effd40039c0 msgr2=0x7effd4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe77fe640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effd40039c0 0x7effd4003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effd8003940 tx=0x7effd801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8029e2467d60b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd8009520 con 0x7effd40039c0 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 10 monclient: dump: 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.512 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd8003bc0 con 0x7effd40039c0 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 4 set_mon_vals no callback set 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 1 -- 172.21.15.138:0/2533770176 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd8005070 con 0x7effd40039c0 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:07.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effe6ffd640 10 monclient: dump: 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 10 monclient: shutdown 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 1 -- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effd40039c0 msgr2=0x7effd4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effd40039c0 0x7effd4003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effd8003940 tx=0x7effd801c420 comp rx=0 tx=0).stop 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 1 -- 172.21.15.138:0/2533770176 shutdown_connections 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 1 --2- 172.21.15.138:0/2533770176 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effd40039c0 0x7effd4003dc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 1 -- 172.21.15.138:0/2533770176 >> 172.21.15.138:0/2533770176 conn(0x7effe8063640 msgr2=0x7effe8063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 1 -- 172.21.15.138:0/2533770176 shutdown_connections 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.500+0000 7effedea8640 1 -- 172.21.15.138:0/2533770176 wait complete. 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient: build_initial_monmap 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient: monmap: 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:07.515 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 librados: starting msgr at 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 librados: starting objecter 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 Processor -- start 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 -- start start 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 librados: setting wanted keys 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 librados: calling monclient init 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient: init 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd40039c0 0x7effe81d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient(hunting): picked mon.noname-b con 0x7effd40039c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:07.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe8140790 0x7effe8131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient(hunting): picked mon.noname-a con 0x7effe8140790 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 0x7effe813e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient(hunting): picked mon.noname-c con 0x7effe8131f40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7effe8134c50 con 0x7effe8140790 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7effe8134ad0 con 0x7effd40039c0 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7effe8134dd0 con 0x7effe8131f40 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effedea8640 10 monclient(hunting): authenticate will time out at 1210.941393s 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 0x7effe813e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe8140790 0x7effe8131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 0x7effe813e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:44310/0 (socket says 172.21.15.138:44310) 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe8140790 0x7effe8131a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:36010/0 (socket says 172.21.15.138:36010) 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 -- 172.21.15.138:0/3540326134 learned_addr learned my addr 172.21.15.138:0/3540326134 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient(hunting): get_auth_request con 0x7effe8131f40 auth_method 0 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe77fe640 1 -- 172.21.15.138:0/3540326134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd40039c0 msgr2=0x7effe81d0ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 10 monclient(hunting): get_auth_request con 0x7effe8140790 auth_method 0 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe77fe640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd40039c0 0x7effe81d0ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:07.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient(hunting): handle_auth_done global_id 22505 payload 824 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 -- 172.21.15.138:0/3540326134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd40039c0 msgr2=0x7effe81d0ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd40039c0 0x7effe81d0ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 -- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe8140790 msgr2=0x7effe8131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe8140790 0x7effe8131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient: _finish_hunting 0 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 monclient: found mon.noname-c 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:07.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effe7fff640 1 -- 172.21.15.138:0/3540326134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe8142640 con 0x7effe8131f40 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.504+0000 7effdffff640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe8140790 0x7effe8131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe7fff640 1 -- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 msgr2=0x7effe813e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe7fff640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 0x7effe813e170 secure :-1 s=READY pgs=5712 cs=0 l=1 rev1=1 crypto rx=0x7effd40057f0 tx=0x7effe000ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 client.?.objecter ms_handle_connect 0x7effe8131f40 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe7fff640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 0x7effe813e170 secure :-1 s=READY pgs=5712 cs=0 l=1 rev1=1 crypto rx=0x7effd40057f0 tx=0x7effe000ed80 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7effe0017440 mon_map magic: 0 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 1 -- 172.21.15.138:0/3540326134 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0017440 con 0x7effe8131f40 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: dump: 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.520 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: _finish_auth 0 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 1 -- 172.21.15.138:0/3540326134 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe00175e0 con 0x7effe8131f40 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 5 monclient: authenticate success, global_id 22505 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 10 monclient: _renew_subs 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:07.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe813fc30 con 0x7effe8131f40 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe67fc640 4 set_mon_vals no callback set 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 10 client.22505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 10 monclient: _renew_subs 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7effe813e880 con 0x7effe8131f40 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 1 librados: init done 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe8135160 con 0x7effe8131f40 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 1 -- 172.21.15.138:0/3540326134 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe00179d0 con 0x7effe8131f40 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:07.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 10 monclient: dump: 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 1 -- 172.21.15.138:0/3540326134 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7effe0015070 con 0x7effe8131f40 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe4ff9640 1 --2- 172.21.15.138:0/3540326134 >> v2:172.21.15.190:6800/1562782189 conn(0x7effc803ce80 0x7effc803f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe77fe640 1 --2- 172.21.15.138:0/3540326134 >> v2:172.21.15.190:6800/1562782189 conn(0x7effc803ce80 0x7effc803f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:07.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.508+0000 7effe77fe640 10 monclient: get_auth_request con 0x7effc803ce80 auth_method 0 2024-09-11T05:12:07.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.512+0000 7effe4ff9640 10 client.22505.objecter ms_handle_connect 0x7effc803ce80 2024-09-11T05:12:07.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.512+0000 7effe77fe640 1 --2- 172.21.15.138:0/3540326134 >> v2:172.21.15.190:6800/1562782189 conn(0x7effc803ce80 0x7effc803f340 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7effd8006fd0 tx=0x7effd8005a90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:07.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.512+0000 7effe4ff9640 1 -- 172.21.15.138:0/3540326134 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(208..208 src has 1..208) ==== 6583+0+0 (secure 0 0 0) 0x7effe00505a0 con 0x7effe8131f40 2024-09-11T05:12:07.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.512+0000 7effe4ff9640 10 client.22505.objecter ms_dispatch 0x7effe8000ba0 osd_map(208..208 src has 1..208) 2024-09-11T05:12:07.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.512+0000 7effe4ff9640 3 client.22505.objecter handle_osd_map got epochs [208,208] > 0 2024-09-11T05:12:07.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.512+0000 7effe4ff9640 3 client.22505.objecter handle_osd_map decoding full epoch 208 2024-09-11T05:12:07.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:07.512+0000 7effe4ff9640 20 client.22505.objecter dump_active .. 0 homeless 2024-09-11T05:12:08.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.336+0000 7effe4ff9640 1 -- 172.21.15.138:0/3540326134 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7effe004e890 con 0x7effe8131f40 2024-09-11T05:12:08.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.336+0000 7effe4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:08.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.336+0000 7effe4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:08.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.576+0000 7effedea8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:12:08.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.576+0000 7effedea8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:08.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.576+0000 7effedea8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:08.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.576+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7effe809e110 con 0x7effe8131f40 2024-09-11T05:12:08.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.576+0000 7effe4ff9640 1 -- 172.21.15.138:0/3540326134 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7effe0020d90 con 0x7effe8131f40 2024-09-11T05:12:08.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.576+0000 7effe4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:08.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.576+0000 7effe4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:08.579 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:12:08.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 10 librados: watch_flush enter 2024-09-11T05:12:08.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 10 librados: watch_flush exit 2024-09-11T05:12:08.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 20 client.22505.objecter shutdown clearing up homeless session... 2024-09-11T05:12:08.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 10 client.22505.objecter successfully canceled tick 2024-09-11T05:12:08.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 >> v2:172.21.15.190:6800/1562782189 conn(0x7effc803ce80 msgr2=0x7effc803f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:08.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 --2- 172.21.15.138:0/3540326134 >> v2:172.21.15.190:6800/1562782189 conn(0x7effc803ce80 0x7effc803f340 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7effd8006fd0 tx=0x7effd8005a90 comp rx=0 tx=0).stop 2024-09-11T05:12:08.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 10 monclient: shutdown 2024-09-11T05:12:08.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 msgr2=0x7effe813e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 0x7effe813e170 secure :-1 s=READY pgs=5712 cs=0 l=1 rev1=1 crypto rx=0x7effd40057f0 tx=0x7effe000ed80 comp rx=0 tx=0).stop 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 shutdown_connections 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 --2- 172.21.15.138:0/3540326134 >> v2:172.21.15.190:6800/1562782189 conn(0x7effc803ce80 0x7effc803f340 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7effe8131f40 0x7effe813e170 unknown :-1 s=CLOSED pgs=5712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7effe8140790 0x7effe8131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 --2- 172.21.15.138:0/3540326134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7effd40039c0 0x7effe81d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 >> 172.21.15.138:0/3540326134 conn(0x7effe8063640 msgr2=0x7effe809ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 shutdown_connections 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 -- 172.21.15.138:0/3540326134 wait complete. 2024-09-11T05:12:08.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:08.580+0000 7effedea8640 1 librados: shutdown 2024-09-11T05:12:11.598 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient: get_monmap_and_config 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient: build_initial_monmap 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient: monmap: 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:11.727154+0000 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:11.727154+0000 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:11.726 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:11.727 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:11.727 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:11.727 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 1 Processor -- start 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 1 -- start start 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient: init 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5a941355e0 0x7f5a941359c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient(hunting): picked mon.noname-c con 0x7f5a941355e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a94134dd0 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:11.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5a94138a60 0x7f5a941406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient(hunting): picked mon.noname-a con 0x7f5a94138a60 addr v2:172.21.15.138:3300/0 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a94074990 con 0x7f5a94134dd0 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f5a940a2d00 con 0x7f5a94138a60 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5a940a2740 con 0x7f5a941355e0 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a98e9c640 10 monclient(hunting): authenticate will time out at 1215.163314s 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a91d9b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f5a94134dd0 0x7f5a94138350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:59862/0 (socket says 172.21.15.138:59862) 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a91d9b640 1 -- 172.21.15.138:0/3806273030 learned_addr learned my addr 172.21.15.138:0/3806273030 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a9259c640 1 -- 172.21.15.138:0/3806273030 >> v2:172.21.15.190:3300/0 conn(0x7f5a941355e0 msgr2=0x7f5a941359c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a9259c640 1 --2- 172.21.15.138:0/3806273030 >> v2:172.21.15.190:3300/0 conn(0x7f5a941355e0 0x7f5a941359c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 1 --2- 172.21.15.138:0/3806273030 >> v2:172.21.15.138:3300/0 conn(0x7f5a94138a60 0x7f5a941406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a9159a640 1 -- 172.21.15.138:0/3806273030 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:12:11.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient(hunting): get_auth_request con 0x7f5a94138a60 auth_method 0 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient(hunting): handle_auth_done global_id 64170 payload 274 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 1 -- 172.21.15.138:0/3806273030 >> v2:172.21.15.190:3300/0 conn(0x7f5a941355e0 msgr2=0x7f5a941359c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 1 --2- 172.21.15.138:0/3806273030 >> v2:172.21.15.190:3300/0 conn(0x7f5a941355e0 0x7f5a941359c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 1 -- 172.21.15.138:0/3806273030 >> v1:172.21.15.138:6789/0 conn(0x7f5a94134dd0 legacy=0x7f5a94138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 1 monclient: found mon.noname-a 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:11.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.724+0000 7f5a92d9d640 1 -- 172.21.15.138:0/3806273030 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a940a2a20 con 0x7f5a94138a60 2024-09-11T05:12:11.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 1 -- 172.21.15.138:0/3806273030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 msgr2=0x7f5a941406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:11.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 1 --2- 172.21.15.138:0/3806273030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a941406f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5a940a4ac0 tx=0x7f5a8c01c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5cfae2d34528e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:11.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 1 -- 172.21.15.138:0/3806273030 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0b2070 con 0x7f5a94138a60 2024-09-11T05:12:11.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: dump: 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:11.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: _finish_auth 0 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: _check_auth_tickets 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 1 -- 172.21.15.138:0/3806273030 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a8c0a5860 con 0x7f5a94138a60 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 1 -- 172.21.15.138:0/3806273030 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0a5c30 con 0x7f5a94138a60 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 5 monclient: authenticate success, global_id 64170 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 4 set_mon_vals no callback set 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9159a640 10 monclient: dump: 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:11.735 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient: shutdown 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- 172.21.15.138:0/3806273030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 msgr2=0x7f5a941406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/3806273030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a941406f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5a940a4ac0 tx=0x7f5a8c01c7e0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- 172.21.15.138:0/3806273030 shutdown_connections 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/3806273030 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a941406f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/3806273030 >> v2:172.21.15.190:3300/0 conn(0x7f5a941355e0 0x7f5a941359c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- 172.21.15.138:0/3806273030 >> 172.21.15.138:0/3806273030 conn(0x7f5a94063640 msgr2=0x7f5a94063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- 172.21.15.138:0/3806273030 shutdown_connections 2024-09-11T05:12:11.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- 172.21.15.138:0/3806273030 wait complete. 2024-09-11T05:12:11.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient: build_initial_monmap 2024-09-11T05:12:11.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient: monmap: 2024-09-11T05:12:11.741 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:11.741 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 librados: starting msgr at 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 librados: starting objecter 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 Processor -- start 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- start start 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 librados: setting wanted keys 2024-09-11T05:12:11.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 librados: calling monclient init 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient: init 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5a94134dd0 0x7f5a941d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient(hunting): picked mon.noname-c con 0x7f5a94134dd0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a941355e0 0x7f5a94140010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient(hunting): picked mon.noname-b con 0x7f5a941355e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a94140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient(hunting): picked mon.noname-a con 0x7f5a94138a60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5a941435d0 con 0x7f5a94138a60 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a94143450 con 0x7f5a941355e0 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:11.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5a94143750 con 0x7f5a94134dd0 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a98e9c640 10 monclient(hunting): authenticate will time out at 1215.171133s 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a94140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5a94134dd0 0x7f5a941d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a94140550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:39580/0 (socket says 172.21.15.138:39580) 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 1 -- 172.21.15.138:0/323352630 learned_addr learned my addr 172.21.15.138:0/323352630 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 10 monclient(hunting): get_auth_request con 0x7f5a94138a60 auth_method 0 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient(hunting): get_auth_request con 0x7f5a94134dd0 auth_method 0 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:11.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a91d9b640 1 -- 172.21.15.138:0/323352630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a941355e0 msgr2=0x7f5a94140010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a91d9b640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a941355e0 0x7f5a94140010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient(hunting): handle_auth_done global_id 22508 payload 824 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 1 -- 172.21.15.138:0/323352630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a941355e0 msgr2=0x7f5a94140010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a941355e0 0x7f5a94140010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 1 -- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 msgr2=0x7f5a94140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:11.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a94140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient: _finish_hunting 0 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 1 monclient: found mon.noname-c 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a9259c640 1 -- 172.21.15.138:0/323352630 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a940a0560 con 0x7f5a94134dd0 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.732+0000 7f5a92d9d640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a94140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a9259c640 1 -- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5a94134dd0 msgr2=0x7f5a941d0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a9259c640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5a94134dd0 0x7f5a941d0cf0 secure :-1 s=READY pgs=5717 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400b470 tx=0x7f5a8400b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 client.?.objecter ms_handle_connect 0x7f5a94134dd0 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 1 -- 172.21.15.138:0/323352630 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a84005070 con 0x7f5a94134dd0 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: dump: 2024-09-11T05:12:11.746 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: _finish_auth 0 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 1 -- 172.21.15.138:0/323352630 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a84015070 con 0x7f5a94134dd0 2024-09-11T05:12:11.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 5 monclient: authenticate success, global_id 22508 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 10 monclient: _renew_subs 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a940a0cb0 con 0x7f5a94134dd0 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 1 -- 172.21.15.138:0/323352630 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a84005450 con 0x7f5a94134dd0 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 monclient: dump: 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:11.748 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a90d99640 4 set_mon_vals no callback set 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 10 client.22508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 10 monclient: _renew_subs 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a941398b0 con 0x7f5a94134dd0 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 1 librados: init done 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a941359c0 con 0x7f5a94134dd0 2024-09-11T05:12:11.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 1 -- 172.21.15.138:0/323352630 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5a840056b0 con 0x7f5a94134dd0 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 1 --2- 172.21.15.138:0/323352630 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5a7403ce80 0x7f5a7403f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 1 -- 172.21.15.138:0/323352630 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(208..208 src has 1..208) ==== 6583+0+0 (secure 0 0 0) 0x7f5a8404f9c0 con 0x7f5a94134dd0 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 10 client.22508.objecter ms_dispatch 0x7f5a94000ba0 osd_map(208..208 src has 1..208) 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 3 client.22508.objecter handle_osd_map got epochs [208,208] > 0 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 3 client.22508.objecter handle_osd_map decoding full epoch 208 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a82ffd640 20 client.22508.objecter dump_active .. 0 homeless 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a91d9b640 1 --2- 172.21.15.138:0/323352630 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5a7403ce80 0x7f5a7403f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.736+0000 7f5a91d9b640 10 monclient: get_auth_request con 0x7f5a7403ce80 auth_method 0 2024-09-11T05:12:11.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.740+0000 7f5a91d9b640 1 --2- 172.21.15.138:0/323352630 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5a7403ce80 0x7f5a7403f340 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f5a9412f760 tx=0x7f5a88007b30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:11.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.740+0000 7f5a82ffd640 1 -- 172.21.15.138:0/323352630 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:12:11.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.744+0000 7f5a82ffd640 10 client.22508.objecter ms_handle_connect 0x7f5a7403ce80 2024-09-11T05:12:11.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.744+0000 7f5a82ffd640 1 -- 172.21.15.138:0/323352630 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f5a8400d040 con 0x7f5a94134dd0 2024-09-11T05:12:11.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.744+0000 7f5a82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:11.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.744+0000 7f5a82ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:11.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.960+0000 7f5a98e9c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:12:11.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.960+0000 7f5a98e9c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:11.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.960+0000 7f5a98e9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:11.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.960+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5a940a1920 con 0x7f5a94134dd0 2024-09-11T05:12:11.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.960+0000 7f5a82ffd640 1 -- 172.21.15.138:0/323352630 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f5a84017d20 con 0x7f5a94134dd0 2024-09-11T05:12:11.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.960+0000 7f5a82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:11.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.960+0000 7f5a82ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:11.963 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:12:11.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 10 librados: watch_flush enter 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 10 librados: watch_flush exit 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 20 client.22508.objecter shutdown clearing up homeless session... 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 10 client.22508.objecter successfully canceled tick 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5a7403ce80 msgr2=0x7f5a7403f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/323352630 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5a7403ce80 0x7f5a7403f340 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f5a9412f760 tx=0x7f5a88007b30 comp rx=0 tx=0).stop 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 10 monclient: shutdown 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5a94134dd0 msgr2=0x7f5a941d0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5a94134dd0 0x7f5a941d0cf0 secure :-1 s=READY pgs=5717 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400b470 tx=0x7f5a8400b930 comp rx=0 tx=0).stop 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 shutdown_connections 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/323352630 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5a7403ce80 0x7f5a7403f340 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5a94138a60 0x7f5a94140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a941355e0 0x7f5a94140010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 --2- 172.21.15.138:0/323352630 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5a94134dd0 0x7f5a941d0cf0 unknown :-1 s=CLOSED pgs=5717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:11.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 >> 172.21.15.138:0/323352630 conn(0x7f5a94063640 msgr2=0x7f5a940a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:11.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 shutdown_connections 2024-09-11T05:12:11.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 -- 172.21.15.138:0/323352630 wait complete. 2024-09-11T05:12:11.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:11.964+0000 7f5a98e9c640 1 librados: shutdown 2024-09-11T05:12:14.981 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:12:15.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient: get_monmap_and_config 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient: build_initial_monmap 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient: monmap: 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:15.110911+0000 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:15.110911+0000 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:15.110 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:15.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 1 Processor -- start 2024-09-11T05:12:15.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 1 -- start start 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient: init 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f157c06e0e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136e30 0x7f157c13b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient(hunting): picked mon.noname-b con 0x7f157c136e30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f157c136170 0x7f157c13da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient(hunting): picked mon.noname-c con 0x7f157c136170 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f157c0a3270 con 0x7f157c06e0e0 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f157c06e4c0 con 0x7f157c136170 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:15.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f157c0a0af0 con 0x7f157c136e30 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1580a71640 10 monclient(hunting): authenticate will time out at 1218.546969s 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f157a59c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f157c06e0e0 0x7f157c136720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:59864/0 (socket says 172.21.15.138:59864) 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f157a59c640 1 -- 172.21.15.138:0/875288779 learned_addr learned my addr 172.21.15.138:0/875288779 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 1 --2- 172.21.15.138:0/875288779 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136e30 0x7f157c13b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f157ad9d640 1 -- 172.21.15.138:0/875288779 >> v2:172.21.15.190:3300/0 conn(0x7f157c136170 msgr2=0x7f157c13da20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient(hunting): get_auth_request con 0x7f157c136e30 auth_method 0 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f157ad9d640 1 --2- 172.21.15.138:0/875288779 >> v2:172.21.15.190:3300/0 conn(0x7f157c136170 0x7f157c13da20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:15.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient(hunting): handle_auth_done global_id 22511 payload 274 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 1 -- 172.21.15.138:0/875288779 >> v2:172.21.15.190:3300/0 conn(0x7f157c136170 msgr2=0x7f157c13da20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 1 --2- 172.21.15.138:0/875288779 >> v2:172.21.15.190:3300/0 conn(0x7f157c136170 0x7f157c13da20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 1 -- 172.21.15.138:0/875288779 >> v1:172.21.15.138:6789/0 conn(0x7f157c06e0e0 legacy=0x7f157c136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 1 monclient: found mon.noname-b 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:15.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.108+0000 7f1579d9b640 1 -- 172.21.15.138:0/875288779 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157c0a2f90 con 0x7f157c136e30 2024-09-11T05:12:15.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.112+0000 7f1579d9b640 1 -- 172.21.15.138:0/875288779 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136e30 msgr2=0x7f157c13b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:15.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.112+0000 7f1579d9b640 1 --2- 172.21.15.138:0/875288779 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136e30 0x7f157c13b4d0 secure :-1 s=READY pgs=5719 cs=0 l=1 rev1=1 crypto rx=0x7f15700097c0 tx=0x7f157001c230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f291596cfeca6b75 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:15.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.112+0000 7f157959a640 1 -- 172.21.15.138:0/875288779 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15700b0070 con 0x7f157c136e30 2024-09-11T05:12:15.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.112+0000 7f157959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:15.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.112+0000 7f157959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:15.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.112+0000 7f157959a640 10 monclient: dump: 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 10 monclient: _finish_auth 0 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 10 monclient: _check_auth_tickets 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:15.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 1 -- 172.21.15.138:0/875288779 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15700ab600 con 0x7f157c136e30 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 5 monclient: authenticate success, global_id 22511 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 4 set_mon_vals no callback set 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 1 -- 172.21.15.138:0/875288779 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15700ab9d0 con 0x7f157c136e30 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157959a640 10 monclient: dump: 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:15.119 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient: shutdown 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- 172.21.15.138:0/875288779 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136e30 msgr2=0x7f157c13b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 --2- 172.21.15.138:0/875288779 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136e30 0x7f157c13b4d0 secure :-1 s=READY pgs=5719 cs=0 l=1 rev1=1 crypto rx=0x7f15700097c0 tx=0x7f157001c230 comp rx=0 tx=0).stop 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- 172.21.15.138:0/875288779 shutdown_connections 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 --2- 172.21.15.138:0/875288779 >> v2:172.21.15.190:3300/0 conn(0x7f157c136170 0x7f157c13da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 --2- 172.21.15.138:0/875288779 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136e30 0x7f157c13b4d0 unknown :-1 s=CLOSED pgs=5719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- 172.21.15.138:0/875288779 >> 172.21.15.138:0/875288779 conn(0x7f157c063640 msgr2=0x7f157c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- 172.21.15.138:0/875288779 shutdown_connections 2024-09-11T05:12:15.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- 172.21.15.138:0/875288779 wait complete. 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient: build_initial_monmap 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient: monmap: 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 librados: starting msgr at 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 librados: starting objecter 2024-09-11T05:12:15.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 Processor -- start 2024-09-11T05:12:15.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- start start 2024-09-11T05:12:15.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 librados: setting wanted keys 2024-09-11T05:12:15.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 librados: calling monclient init 2024-09-11T05:12:15.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient: init 2024-09-11T05:12:15.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f157c06e0e0 0x7f157c12f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient(hunting): picked mon.noname-b con 0x7f157c06e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136170 0x7f157c12ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient(hunting): picked mon.noname-c con 0x7f157c136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f157c136e30 0x7f157c1d03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient(hunting): picked mon.noname-a con 0x7f157c136e30 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f157c140b40 con 0x7f157c136e30 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f157c1409c0 con 0x7f157c06e0e0 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f157c140cc0 con 0x7f157c136170 2024-09-11T05:12:15.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1580a71640 10 monclient(hunting): authenticate will time out at 1218.554272s 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157ad9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f157c136e30 0x7f157c1d03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136170 0x7f157c12ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136170 0x7f157c12ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:51400/0 (socket says 172.21.15.138:51400) 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 -- 172.21.15.138:0/1295311182 learned_addr learned my addr 172.21.15.138:0/1295311182 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157ad9d640 10 monclient(hunting): get_auth_request con 0x7f157c136e30 auth_method 0 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157a59c640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f157c06e0e0 msgr2=0x7f157c12f9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient(hunting): get_auth_request con 0x7f157c136170 auth_method 0 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:15.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157a59c640 1 --2- 172.21.15.138:0/1295311182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f157c06e0e0 0x7f157c12f9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f157ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient(hunting): handle_auth_done global_id 22514 payload 824 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 -- 172.21.15.138:0/1295311182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f157c06e0e0 msgr2=0x7f157c12f9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 --2- 172.21.15.138:0/1295311182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f157c06e0e0 0x7f157c12f9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 -- 172.21.15.138:0/1295311182 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f157c136e30 msgr2=0x7f157c1d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 --2- 172.21.15.138:0/1295311182 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f157c136e30 0x7f157c1d03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:12:15.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 monclient: found mon.noname-c 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.116+0000 7f1579d9b640 1 -- 172.21.15.138:0/1295311182 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157c0a0110 con 0x7f157c136170 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1579d9b640 1 -- 172.21.15.138:0/1295311182 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136170 msgr2=0x7f157c12ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1579d9b640 1 --2- 172.21.15.138:0/1295311182 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136170 0x7f157c12ff20 secure :-1 s=READY pgs=5720 cs=0 l=1 rev1=1 crypto rx=0x7f157001cad0 tx=0x7f1570018fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 client.?.objecter ms_handle_connect 0x7f157c136170 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 1 -- 172.21.15.138:0/1295311182 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15700bb070 con 0x7f157c136170 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: dump: 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:15.129 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: _finish_auth 0 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: _check_auth_tickets 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 5 monclient: authenticate success, global_id 22514 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 1 -- 172.21.15.138:0/1295311182 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15700b7aa0 con 0x7f157c136170 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 10 monclient: _renew_subs 2024-09-11T05:12:15.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f157c13b820 con 0x7f157c136170 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 1 -- 172.21.15.138:0/1295311182 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15700b0040 con 0x7f157c136170 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 monclient: dump: 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:15.131 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 10 client.22514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1578d99640 4 set_mon_vals no callback set 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 10 monclient: _renew_subs 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f157c13d020 con 0x7f157c136170 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 1 librados: init done 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 1 -- 172.21.15.138:0/1295311182 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f15700b5430 con 0x7f157c136170 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f157c136720 con 0x7f157c136170 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 1 --2- 172.21.15.138:0/1295311182 >> v2:172.21.15.190:6800/1562782189 conn(0x7f155c03cea0 0x7f155c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:15.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 1 -- 172.21.15.138:0/1295311182 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(208..208 src has 1..208) ==== 6583+0+0 (secure 0 0 0) 0x7f15700eaaa0 con 0x7f157c136170 2024-09-11T05:12:15.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 10 client.22514.objecter ms_dispatch 0x7f157c000ba0 osd_map(208..208 src has 1..208) 2024-09-11T05:12:15.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 3 client.22514.objecter handle_osd_map got epochs [208,208] > 0 2024-09-11T05:12:15.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 3 client.22514.objecter handle_osd_map decoding full epoch 208 2024-09-11T05:12:15.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f156affd640 20 client.22514.objecter dump_active .. 0 homeless 2024-09-11T05:12:15.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f157a59c640 1 --2- 172.21.15.138:0/1295311182 >> v2:172.21.15.190:6800/1562782189 conn(0x7f155c03cea0 0x7f155c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:15.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.120+0000 7f157a59c640 10 monclient: get_auth_request con 0x7f155c03cea0 auth_method 0 2024-09-11T05:12:15.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.124+0000 7f157a59c640 1 --2- 172.21.15.138:0/1295311182 >> v2:172.21.15.190:6800/1562782189 conn(0x7f155c03cea0 0x7f155c03f360 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f156c004820 tx=0x7f156c007b30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:15.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.124+0000 7f156affd640 10 client.22514.objecter ms_handle_connect 0x7f155c03cea0 2024-09-11T05:12:15.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.124+0000 7f156affd640 1 -- 172.21.15.138:0/1295311182 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f157c13b820 con 0x7f157c136170 2024-09-11T05:12:15.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.124+0000 7f156affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:15.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.124+0000 7f156affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:15.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.348+0000 7f1580a71640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:12:15.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.348+0000 7f1580a71640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:15.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.348+0000 7f1580a71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:15.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.348+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f157c0a6870 con 0x7f157c136170 2024-09-11T05:12:15.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.348+0000 7f156affd640 1 -- 172.21.15.138:0/1295311182 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f157c0a3270 con 0x7f157c136170 2024-09-11T05:12:15.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.348+0000 7f156affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:15.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.348+0000 7f156affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:15.351 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 10 librados: watch_flush enter 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 10 librados: watch_flush exit 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 20 client.22514.objecter shutdown clearing up homeless session... 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 10 client.22514.objecter successfully canceled tick 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 >> v2:172.21.15.190:6800/1562782189 conn(0x7f155c03cea0 msgr2=0x7f155c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 --2- 172.21.15.138:0/1295311182 >> v2:172.21.15.190:6800/1562782189 conn(0x7f155c03cea0 0x7f155c03f360 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f156c004820 tx=0x7f156c007b30 comp rx=0 tx=0).stop 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 10 monclient: shutdown 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136170 msgr2=0x7f157c12ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 --2- 172.21.15.138:0/1295311182 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136170 0x7f157c12ff20 secure :-1 s=READY pgs=5720 cs=0 l=1 rev1=1 crypto rx=0x7f157001cad0 tx=0x7f1570018fd0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 shutdown_connections 2024-09-11T05:12:15.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 --2- 172.21.15.138:0/1295311182 >> v2:172.21.15.190:6800/1562782189 conn(0x7f155c03cea0 0x7f155c03f360 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 --2- 172.21.15.138:0/1295311182 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f157c136e30 0x7f157c1d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 --2- 172.21.15.138:0/1295311182 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f157c136170 0x7f157c12ff20 unknown :-1 s=CLOSED pgs=5720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 --2- 172.21.15.138:0/1295311182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f157c06e0e0 0x7f157c12f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:15.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 >> 172.21.15.138:0/1295311182 conn(0x7f157c063640 msgr2=0x7f157c132b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:15.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 shutdown_connections 2024-09-11T05:12:15.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 -- 172.21.15.138:0/1295311182 wait complete. 2024-09-11T05:12:15.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:15.352+0000 7f1580a71640 1 librados: shutdown 2024-09-11T05:12:18.369 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:12:18.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient: get_monmap_and_config 2024-09-11T05:12:18.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient: build_initial_monmap 2024-09-11T05:12:18.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient: monmap: 2024-09-11T05:12:18.497 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:18.498647+0000 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:18.498647+0000 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:18.498 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:18.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 1 Processor -- start 2024-09-11T05:12:18.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 1 -- start start 2024-09-11T05:12:18.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient: init 2024-09-11T05:12:18.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 0x7f39f8136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient(hunting): picked mon.noname-c con 0x7f39f806e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39f8136170 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient(hunting): picked mon.noname-b con 0x7f39f8136c60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f80a3270 con 0x7f39f8136170 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f39f806e4c0 con 0x7f39f806e0e0 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f39f80a0af0 con 0x7f39f8136c60 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39fd111640 10 monclient(hunting): authenticate will time out at 1221.935272s 2024-09-11T05:12:18.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f6d9d640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 msgr2=0x7f39f8136720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f659c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f39f8136170 0x7f39f80a1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:44160/0 (socket says 172.21.15.138:44160) 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 10 monclient(hunting): get_auth_request con 0x7f39f8136c60 auth_method 0 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f659c640 1 -- 172.21.15.138:0/2647253105 learned_addr learned my addr 172.21.15.138:0/2647253105 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 0x7f39f8136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:12:18.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.496+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:18.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.500+0000 7f39f659c640 1 -- 172.21.15.138:0/2647253105 >> v1:172.21.15.138:6789/0 conn(0x7f39f8136170 legacy=0x7f39f80a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:18.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f6d9d640 1 -- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 msgr2=0x7f39f8136720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 0x7f39f8136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 10 monclient(hunting): get_auth_request con 0x7f39f8136c60 auth_method 0 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:12:18.702 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:18.700+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:12:19.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:19.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 10 monclient(hunting): get_auth_request con 0x7f39f8136c60 auth_method 0 2024-09-11T05:12:19.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:19.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:19.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:12:19.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:12:19.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:12:19.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:12:19.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:12:19.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f6d9d640 1 -- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 msgr2=0x7f39f8136720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:19.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.104+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 0x7f39f8136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f6d9d640 1 -- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 msgr2=0x7f39f8136720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 0x7f39f8136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 10 monclient(hunting): get_auth_request con 0x7f39f8136c60 auth_method 0 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:12:19.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:12:19.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:19.908+0000 7f39f759e640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:12:21.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 10 monclient(hunting): tick 2024-09-11T05:12:21.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 monclient(hunting): continuing hunt 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 msgr2=0x7f39f80a2030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39f8136c60 0x7f39f80a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 -- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 msgr2=0x7f39f8136720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 --2- 172.21.15.138:0/2647253105 >> v2:172.21.15.190:3300/0 conn(0x7f39f806e0e0 0x7f39f8136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 -- 172.21.15.138:0/2647253105 >> v1:172.21.15.138:6789/0 conn(0x7f39f8136170 legacy=0x7f39f80a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 10 monclient: _add_conns ranks=[3,1] 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 --2- 172.21.15.138:0/2647253105 >> v2:172.21.15.138:3300/0 conn(0x7f39e40037a0 0x7f39e4005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 10 monclient(hunting): picked mon.noname-a con 0x7f39e40037a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39e40062f0 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 -- 172.21.15.138:0/2647253105 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f80a2f90 con 0x7f39e40062f0 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 1 -- 172.21.15.138:0/2647253105 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f39f806e4c0 con 0x7f39e40037a0 2024-09-11T05:12:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f4d99640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f659c640 1 -- 172.21.15.138:0/2647253105 >> v1:172.21.15.190:6789/0 conn(0x7f39e40062f0 legacy=0x7f39e40087b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:12:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/2647253105 >> v2:172.21.15.138:3300/0 conn(0x7f39e40037a0 0x7f39e4005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f6d9d640 10 monclient(hunting): get_auth_request con 0x7f39e40037a0 auth_method 0 2024-09-11T05:12:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.496+0000 7f39f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 10 monclient(hunting): handle_auth_done global_id 64179 payload 274 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 1 -- 172.21.15.138:0/2647253105 >> v1:172.21.15.190:6789/0 conn(0x7f39e40062f0 legacy=0x7f39e40087b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 1 monclient: found mon.noname-a 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.500+0000 7f39f6d9d640 1 -- 172.21.15.138:0/2647253105 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e400b0b0 con 0x7f39e40037a0 2024-09-11T05:12:21.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f6d9d640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e40037a0 msgr2=0x7f39e4005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:21.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e40037a0 0x7f39e4005c00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f39ec003940 tx=0x7f39ec01cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a12d7052d7a547d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 1 -- 172.21.15.138:0/2647253105 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ec0a59e0 con 0x7f39e40037a0 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: dump: 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: _finish_auth 0 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: _check_auth_tickets 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 1 -- 172.21.15.138:0/2647253105 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39ec0a9070 con 0x7f39e40037a0 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 1 -- 172.21.15.138:0/2647253105 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ec0a5de0 con 0x7f39e40037a0 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39f5d9b640 10 monclient: dump: 2024-09-11T05:12:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39fd111640 5 monclient: authenticate success, global_id 64179 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39fd111640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.508+0000 7f39fd111640 4 set_mon_vals no callback set 2024-09-11T05:12:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 10 monclient: shutdown 2024-09-11T05:12:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 1 -- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e40037a0 msgr2=0x7f39e4005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e40037a0 0x7f39e4005c00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f39ec003940 tx=0x7f39ec01cb40 comp rx=0 tx=0).stop 2024-09-11T05:12:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39f759e640 1 -- 172.21.15.138:0/2647253105 reap_dead start 2024-09-11T05:12:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 1 -- 172.21.15.138:0/2647253105 shutdown_connections 2024-09-11T05:12:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 1 --2- 172.21.15.138:0/2647253105 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39e40037a0 0x7f39e4005c00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 1 -- 172.21.15.138:0/2647253105 >> 172.21.15.138:0/2647253105 conn(0x7f39f8063640 msgr2=0x7f39f8063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 1 -- 172.21.15.138:0/2647253105 shutdown_connections 2024-09-11T05:12:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 1 -- 172.21.15.138:0/2647253105 wait complete. 2024-09-11T05:12:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 10 monclient: build_initial_monmap 2024-09-11T05:12:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.512+0000 7f39fd111640 10 monclient: monmap: 2024-09-11T05:12:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:21.516 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.516+0000 7f39fd111640 1 librados: starting msgr at 2024-09-11T05:12:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.516+0000 7f39fd111640 1 librados: starting objecter 2024-09-11T05:12:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.516+0000 7f39fd111640 1 Processor -- start 2024-09-11T05:12:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 -- start start 2024-09-11T05:12:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 librados: setting wanted keys 2024-09-11T05:12:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 librados: calling monclient init 2024-09-11T05:12:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient: init 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39e40037a0 0x7f39f813d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient(hunting): picked mon.noname-c con 0x7f39e40037a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39f80728b0 0x7f39f8070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient(hunting): picked mon.noname-a con 0x7f39f80728b0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f8072cb0 0x7f39f8071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient(hunting): picked mon.noname-b con 0x7f39f8072cb0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81410a0 con 0x7f39f80728b0 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f39f8140f20 con 0x7f39f8072cb0 2024-09-11T05:12:21.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f39f8141220 con 0x7f39e40037a0 2024-09-11T05:12:21.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:21.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39fd111640 10 monclient(hunting): authenticate will time out at 1224.961919s 2024-09-11T05:12:21.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f759e640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f8072cb0 msgr2=0x7f39f8071430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:21.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f759e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f8072cb0 0x7f39f8071430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:21.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39f80728b0 0x7f39f8070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39f80728b0 0x7f39f8070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46060/0 (socket says 172.21.15.138:46060) 2024-09-11T05:12:21.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 -- 172.21.15.138:0/575661477 learned_addr learned my addr 172.21.15.138:0/575661477 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:21.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient(hunting): get_auth_request con 0x7f39f80728b0 auth_method 0 2024-09-11T05:12:21.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39e40037a0 0x7f39f813d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f6d9d640 10 monclient(hunting): get_auth_request con 0x7f39e40037a0 auth_method 0 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient(hunting): handle_auth_done global_id 64182 payload 824 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 -- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39e40037a0 msgr2=0x7f39f813d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39e40037a0 0x7f39f813d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 -- 172.21.15.138:0/575661477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f8072cb0 msgr2=0x7f39f8071430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f8072cb0 0x7f39f8071430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient: _finish_hunting 0 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 monclient: found mon.noname-a 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f659c640 1 -- 172.21.15.138:0/575661477 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f8138130 con 0x7f39f80728b0 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.524+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39e40037a0 0x7f39f813d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:12:21.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.528+0000 7f39df7fe640 10 client.?.objecter ms_handle_connect 0x7f39f80728b0 2024-09-11T05:12:21.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.528+0000 7f39df7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:21.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.528+0000 7f39f659c640 1 -- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39f80728b0 msgr2=0x7f39f8070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:21.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.528+0000 7f39f659c640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39f80728b0 0x7f39f8070ef0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c400 tx=0x7f39e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:21.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 1 -- 172.21.15.138:0/575661477 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e8017070 con 0x7f39f80728b0 2024-09-11T05:12:21.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:21.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: dump: 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: _finish_auth 0 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 5 monclient: authenticate success, global_id 64182 2024-09-11T05:12:21.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 10 monclient: _renew_subs 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f8138410 con 0x7f39f80728b0 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 10 client.64182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 10 monclient: _renew_subs 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f81378f0 con 0x7f39f80728b0 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 1 librados: init done 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 1 -- 172.21.15.138:0/575661477 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e80070d0 con 0x7f39f80728b0 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 1 -- 172.21.15.138:0/575661477 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e8021060 con 0x7f39f80728b0 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39df7fe640 10 monclient: dump: 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.535 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39f4d99640 4 set_mon_vals no callback set 2024-09-11T05:12:21.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:21.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:21.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:21.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.532+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d8005090 con 0x7f39f80728b0 2024-09-11T05:12:21.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.536+0000 7f39df7fe640 1 -- 172.21.15.138:0/575661477 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f39e8007300 con 0x7f39f80728b0 2024-09-11T05:12:21.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.536+0000 7f39df7fe640 1 --2- 172.21.15.138:0/575661477 >> v2:172.21.15.190:6800/1562782189 conn(0x7f39e403e090 0x7f39e4040550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.536+0000 7f39df7fe640 1 -- 172.21.15.138:0/575661477 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 6213+0+0 (secure 0 0 0) 0x7f39e804ef80 con 0x7f39f80728b0 2024-09-11T05:12:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.536+0000 7f39df7fe640 10 client.64182.objecter ms_dispatch 0x7f39f8000ba0 osd_map(209..209 src has 1..209) 2024-09-11T05:12:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.536+0000 7f39df7fe640 3 client.64182.objecter handle_osd_map got epochs [209,209] > 0 2024-09-11T05:12:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.536+0000 7f39df7fe640 3 client.64182.objecter handle_osd_map decoding full epoch 209 2024-09-11T05:12:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.536+0000 7f39df7fe640 20 client.64182.objecter dump_active .. 0 homeless 2024-09-11T05:12:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.540+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/575661477 >> v2:172.21.15.190:6800/1562782189 conn(0x7f39e403e090 0x7f39e4040550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.540+0000 7f39f6d9d640 10 monclient: get_auth_request con 0x7f39e403e090 auth_method 0 2024-09-11T05:12:21.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.544+0000 7f39df7fe640 10 client.64182.objecter ms_handle_connect 0x7f39e403e090 2024-09-11T05:12:21.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.544+0000 7f39f6d9d640 1 --2- 172.21.15.138:0/575661477 >> v2:172.21.15.190:6800/1562782189 conn(0x7f39e403e090 0x7f39e4040550 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7f39ec0aa000 tx=0x7f39ec0a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:21.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.556+0000 7f39df7fe640 1 -- 172.21.15.138:0/575661477 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f39f80a3270 con 0x7f39f80728b0 2024-09-11T05:12:21.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.556+0000 7f39df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:21.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.556+0000 7f39df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:21.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:12:21.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:21.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:21.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f39d8005380 con 0x7f39f80728b0 2024-09-11T05:12:21.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39df7fe640 1 -- 172.21.15.138:0/575661477 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f39e801a360 con 0x7f39f80728b0 2024-09-11T05:12:21.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:12:21.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:21.802 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 10 librados: watch_flush enter 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 10 librados: watch_flush exit 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 20 client.64182.objecter shutdown clearing up homeless session... 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 10 client.64182.objecter successfully canceled tick 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 >> v2:172.21.15.190:6800/1562782189 conn(0x7f39e403e090 msgr2=0x7f39e4040550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 --2- 172.21.15.138:0/575661477 >> v2:172.21.15.190:6800/1562782189 conn(0x7f39e403e090 0x7f39e4040550 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7f39ec0aa000 tx=0x7f39ec0a7070 comp rx=0 tx=0).stop 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 10 monclient: shutdown 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:21.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39f80728b0 msgr2=0x7f39f8070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39f80728b0 0x7f39f8070ef0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f39e800c400 tx=0x7f39e800c8c0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 shutdown_connections 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 --2- 172.21.15.138:0/575661477 >> v2:172.21.15.190:6800/1562782189 conn(0x7f39e403e090 0x7f39e4040550 unknown :-1 s=CLOSED pgs=3372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f8072cb0 0x7f39f8071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f39f80728b0 0x7f39f8070ef0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.800+0000 7f39fd111640 1 --2- 172.21.15.138:0/575661477 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f39e40037a0 0x7f39f813d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.804+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 >> 172.21.15.138:0/575661477 conn(0x7f39f8063640 msgr2=0x7f39f80a4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.804+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 shutdown_connections 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.804+0000 7f39fd111640 1 -- 172.21.15.138:0/575661477 wait complete. 2024-09-11T05:12:21.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.804+0000 7f39fd111640 1 librados: shutdown 2024-09-11T05:12:21.820 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:12:21.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T05:12:21.821 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:12:21.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient: get_monmap_and_config 2024-09-11T05:12:21.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient: build_initial_monmap 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient: monmap: 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:21.937945+0000 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:21.937945+0000 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:21.937 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 1 Processor -- start 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 1 -- start start 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient: init 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5ea409da20 0x7f5ea40a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient(hunting): picked mon.noname-c con 0x7f5ea409da20 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ea40a1e60 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4136910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient(hunting): picked mon.noname-b con 0x7f5ea40a22d0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:21.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ea4064590 con 0x7f5ea40a1e60 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5ea409de00 con 0x7f5ea409da20 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5ea40a2700 con 0x7f5ea40a22d0 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5eab0e0640 10 monclient(hunting): authenticate will time out at 1225.374116s 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea3fff640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7f5ea40a1e60 legacy=0x7f5ea4136200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea8e7c640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7f5ea409da20 msgr2=0x7f5ea40a1920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea8e7c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5ea409da20 0x7f5ea40a1920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4136910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:39188/0 (socket says 172.21.15.138:39188) 2024-09-11T05:12:21.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 1 -- 172.21.15.138:0/1232918832 learned_addr learned my addr 172.21.15.138:0/1232918832 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient(hunting): get_auth_request con 0x7f5ea40a22d0 auth_method 0 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient(hunting): handle_auth_done global_id 22526 payload 274 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 1 -- 172.21.15.138:0/1232918832 >> v2:172.21.15.190:3300/0 conn(0x7f5ea409da20 msgr2=0x7f5ea40a1920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 1 --2- 172.21.15.138:0/1232918832 >> v2:172.21.15.190:3300/0 conn(0x7f5ea409da20 0x7f5ea40a1920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 1 -- 172.21.15.138:0/1232918832 >> v1:172.21.15.190:6789/0 conn(0x7f5ea40a1e60 legacy=0x7f5ea4136200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient: _finish_hunting 0 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 1 monclient: found mon.noname-b 2024-09-11T05:12:21.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:21.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:21.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.936+0000 7f5ea967d640 1 -- 172.21.15.138:0/1232918832 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea40a2c70 con 0x7f5ea40a22d0 2024-09-11T05:12:21.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea967d640 1 -- 172.21.15.138:0/1232918832 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 msgr2=0x7f5ea4136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea967d640 1 --2- 172.21.15.138:0/1232918832 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4136910 secure :-1 s=READY pgs=5725 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009e50 tx=0x7f5e9c01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc5f4cb1b7ce0fde server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 1 -- 172.21.15.138:0/1232918832 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0a5710 con 0x7f5ea40a22d0 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: dump: 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.947 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: _finish_auth 0 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: _check_auth_tickets 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 1 -- 172.21.15.138:0/1232918832 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e9c0a58b0 con 0x7f5ea40a22d0 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 5 monclient: authenticate success, global_id 22526 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 1 -- 172.21.15.138:0/1232918832 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0a5c80 con 0x7f5ea40a22d0 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 4 set_mon_vals no callback set 2024-09-11T05:12:21.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5ea37fe640 10 monclient: dump: 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 10 monclient: shutdown 2024-09-11T05:12:21.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 1 -- 172.21.15.138:0/1232918832 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 msgr2=0x7f5ea4136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 1 --2- 172.21.15.138:0/1232918832 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4136910 secure :-1 s=READY pgs=5725 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009e50 tx=0x7f5e9c01c910 comp rx=0 tx=0).stop 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 1 -- 172.21.15.138:0/1232918832 shutdown_connections 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 1 --2- 172.21.15.138:0/1232918832 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4136910 unknown :-1 s=CLOSED pgs=5725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 1 --2- 172.21.15.138:0/1232918832 >> v2:172.21.15.190:3300/0 conn(0x7f5ea409da20 0x7f5ea40a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 1 -- 172.21.15.138:0/1232918832 >> 172.21.15.138:0/1232918832 conn(0x7f5ea4063640 msgr2=0x7f5ea4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.940+0000 7f5eab0e0640 1 -- 172.21.15.138:0/1232918832 shutdown_connections 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 -- 172.21.15.138:0/1232918832 wait complete. 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient: build_initial_monmap 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient: monmap: 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:21.950 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 librados: starting msgr at 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 librados: starting objecter 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 Processor -- start 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 -- start start 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 librados: setting wanted keys 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 librados: calling monclient init 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient: init 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:21.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ea409da20 0x7f5ea40714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient(hunting): picked mon.noname-b con 0x7f5ea409da20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea40a1e60 0x7f5ea4071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient(hunting): picked mon.noname-a con 0x7f5ea40a1e60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4071f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient(hunting): picked mon.noname-c con 0x7f5ea40a22d0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea4140ee0 con 0x7f5ea40a1e60 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea4140d60 con 0x7f5ea409da20 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5ea4141060 con 0x7f5ea40a22d0 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:21.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5eab0e0640 10 monclient(hunting): authenticate will time out at 1225.381537s 2024-09-11T05:12:21.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea8e7c640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ea409da20 msgr2=0x7f5ea40714f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:21.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea3fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea40a1e60 0x7f5ea4071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea8e7c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ea409da20 0x7f5ea40714f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:21.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4071f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:39196/0 (socket says 172.21.15.138:39196) 2024-09-11T05:12:21.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 -- 172.21.15.138:0/3758968284 learned_addr learned my addr 172.21.15.138:0/3758968284 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient(hunting): get_auth_request con 0x7f5ea40a22d0 auth_method 0 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea3fff640 10 monclient(hunting): get_auth_request con 0x7f5ea40a1e60 auth_method 0 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:21.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient(hunting): handle_auth_done global_id 22529 payload 824 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ea409da20 msgr2=0x7f5ea40714f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ea409da20 0x7f5ea40714f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea40a1e60 msgr2=0x7f5ea4071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea40a1e60 0x7f5ea4071a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient: _finish_hunting 0 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 monclient: found mon.noname-c 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:12:21.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.944+0000 7f5ea967d640 1 -- 172.21.15.138:0/3758968284 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea4136af0 con 0x7f5ea40a22d0 2024-09-11T05:12:21.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea967d640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 msgr2=0x7f5ea4071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:21.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea967d640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4071f70 secure :-1 s=READY pgs=5726 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009e50 tx=0x7f5e9c0196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:21.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 client.?.objecter ms_handle_connect 0x7f5ea40a22d0 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0bc020 con 0x7f5ea40a22d0 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: dump: 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.960 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: _finish_auth 0 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: _check_auth_tickets 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e9c0b8880 con 0x7f5ea40a22d0 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 5 monclient: authenticate success, global_id 22529 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 10 monclient: _renew_subs 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ea409e110 con 0x7f5ea40a22d0 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0b8c70 con 0x7f5ea40a22d0 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 monclient: dump: 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea2ffd640 4 set_mon_vals no callback set 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 10 client.22529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 10 monclient: _renew_subs 2024-09-11T05:12:21.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ea4137d70 con 0x7f5ea40a22d0 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 1 librados: init done 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41ce320 0x7f5ea41ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea41348a0 con 0x7f5ea41ce320 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41ce320 0x7f5ea41ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 10 monclient: get_auth_request con 0x7f5ea41ce320 auth_method 0 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:21.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5e9c0c9060 con 0x7f5ea40a22d0 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 10 monclient(hunting): handle_auth_done global_id 64191 payload 824 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 1 -- 172.21.15.138:0/3758968284 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ea4137940 con 0x7f5ea41ce320 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 1 --2- 172.21.15.138:0/3758968284 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5e8403ce50 0x7f5e8403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(209..209 src has 1..209) ==== 6213+0+0 (secure 0 0 0) 0x7f5e9c0eafa0 con 0x7f5ea40a22d0 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 client.22529.objecter ms_dispatch 0x7f5ea4000ba0 osd_map(209..209 src has 1..209) 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 3 client.22529.objecter handle_osd_map got epochs [209,209] > 0 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 3 client.22529.objecter handle_osd_map decoding full epoch 209 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 20 client.22529.objecter dump_active .. 0 homeless 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea3fff640 1 --2- 172.21.15.138:0/3758968284 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5e8403ce50 0x7f5e8403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea3fff640 10 monclient: get_auth_request con 0x7f5e8403ce50 auth_method 0 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 client.22529.objecter ms_handle_connect 0x7f5ea41ce320 2024-09-11T05:12:21.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea17fa640 10 client.22529.objecter resend_mon_ops 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41ce320 msgr2=0x7f5ea41ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.948+0000 7f5ea8e7c640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41ce320 0x7f5ea41ce700 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b450 tx=0x7f5e8c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c005270 con 0x7f5ea41ce320 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 10 monclient: dump: 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.965 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5e8c015070 con 0x7f5ea41ce320 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41ce320 msgr2=0x7f5ea41ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41ce320 0x7f5ea41ce700 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b450 tx=0x7f5e8c00b910 comp rx=0 tx=0).stop 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41ce320 msgr2=0x7f5ea41ce700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41ce320 0x7f5ea41ce700 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b450 tx=0x7f5e8c00b910 comp rx=0 tx=0).stop 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea3fff640 1 --2- 172.21.15.138:0/3758968284 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5e8403ce50 0x7f5e8403f310 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f5e940093d0 tx=0x7f5e94009360 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:21.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.952+0000 7f5ea17fa640 10 client.22529.objecter ms_handle_connect 0x7f5e8403ce50 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5eab0e0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5eab0e0640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41cec40 0x7f5ea412d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5eab0e0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea40a19a0 con 0x7f5ea41cec40 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41cec40 0x7f5ea412d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 10 monclient: get_auth_request con 0x7f5ea41cec40 auth_method 0 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 10 monclient(hunting): handle_auth_done global_id 64194 payload 824 2024-09-11T05:12:21.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.956+0000 7f5ea8e7c640 1 -- 172.21.15.138:0/3758968284 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ea4073a50 con 0x7f5ea41cec40 2024-09-11T05:12:21.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 10 client.22529.objecter ms_handle_connect 0x7f5ea41cec40 2024-09-11T05:12:21.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea8e7c640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41cec40 msgr2=0x7f5ea412d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:21.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 10 client.22529.objecter resend_mon_ops 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea8e7c640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41cec40 0x7f5ea412d880 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b450 tx=0x7f5e8c005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c010480 con 0x7f5ea41cec40 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 10 monclient: dump: 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:21.969 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f5e8c010480 con 0x7f5ea41cec40 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41cec40 msgr2=0x7f5ea412d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41cec40 0x7f5ea412d880 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b450 tx=0x7f5e8c005a30 comp rx=0 tx=0).stop 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41cec40 msgr2=0x7f5ea412d880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea17fa640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5ea41cec40 0x7f5ea412d880 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b450 tx=0x7f5e8c005a30 comp rx=0 tx=0).stop 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:12:21.970 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 24, 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 45590, 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:12:21.971 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:12:21.972 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:12:21.973 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:12:21.974 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:12:21.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:21.976 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:21.977 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:12:21.978 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:21.979 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:12:21.980 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:21.981 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:21.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:21.982 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:12:21.982 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:12:21.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 10 librados: watch_flush enter 2024-09-11T05:12:21.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 10 librados: watch_flush exit 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 20 client.22529.objecter shutdown clearing up homeless session... 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 10 client.22529.objecter successfully canceled tick 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5e8403ce50 msgr2=0x7f5e8403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 --2- 172.21.15.138:0/3758968284 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5e8403ce50 0x7f5e8403f310 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f5e940093d0 tx=0x7f5e94009360 comp rx=0 tx=0).stop 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 10 monclient: shutdown 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 msgr2=0x7f5ea4071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 --2- 172.21.15.138:0/3758968284 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5ea40a22d0 0x7f5ea4071f70 secure :-1 s=READY pgs=5726 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009e50 tx=0x7f5e9c0196b0 comp rx=0 tx=0).stop 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea967d640 1 -- 172.21.15.138:0/3758968284 reap_dead start 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5ea967d640 1 -- 172.21.15.138:0/3758968284 reap_dead start 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 shutdown_connections 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 >> 172.21.15.138:0/3758968284 conn(0x7f5ea4063640 msgr2=0x7f5ea4134e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:21.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 shutdown_connections 2024-09-11T05:12:21.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 -- 172.21.15.138:0/3758968284 wait complete. 2024-09-11T05:12:21.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:21.960+0000 7f5eab0e0640 1 librados: shutdown 2024-09-11T05:12:21.985 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:12:21.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T05:12:21.985 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:12:22.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient: get_monmap_and_config 2024-09-11T05:12:22.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient: build_initial_monmap 2024-09-11T05:12:22.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient: monmap: 2024-09-11T05:12:22.100 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:22.100 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:22.100 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:12:22.101528+0000 2024-09-11T05:12:22.100 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:12:22.101528+0000 2024-09-11T05:12:22.100 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:22.101 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:22.101 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:12:22.101 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:12:22.101 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:12:22.101 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:12:22.101 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:12:22.101 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 1 Processor -- start 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 1 -- start start 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient: init 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient(hunting): picked mon.noname-a con 0x7ff690070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff69009e6e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff69009eb10 addr v1:172.21.15.190:6789/0 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6901390e0 con 0x7ff69009e6e0 2024-09-11T05:12:22.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6900a8990 con 0x7ff69009eb10 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7ff6900703f0 con 0x7ff690070010 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff695678640 10 monclient(hunting): authenticate will time out at 1225.537690s 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46102/0 (socket says 172.21.15.138:46102) 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 1 -- 172.21.15.138:0/1170476618 learned_addr learned my addr 172.21.15.138:0/1170476618 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 10 monclient(hunting): get_auth_request con 0x7ff690070010 auth_method 0 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:22.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68f7fe640 1 -- 172.21.15.138:0/1170476618 >> v1:172.21.15.190:6789/0 conn(0x7ff69009eb10 legacy=0x7ff6901389d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:12:22.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:22.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:22.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.100+0000 7ff68effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:22.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68e7fc640 1 -- 172.21.15.138:0/1170476618 >> v1:172.21.15.138:6789/0 conn(0x7ff69009e6e0 legacy=0x7ff6901382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:22.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 10 monclient(hunting): handle_auth_done global_id 64197 payload 274 2024-09-11T05:12:22.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 1 -- 172.21.15.138:0/1170476618 >> v1:172.21.15.190:6789/0 conn(0x7ff69009eb10 legacy=0x7ff6901389d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:22.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 1 -- 172.21.15.138:0/1170476618 >> v1:172.21.15.138:6789/0 conn(0x7ff69009e6e0 legacy=0x7ff6901382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:22.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 10 monclient: _finish_hunting 0 2024-09-11T05:12:22.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 1 monclient: found mon.noname-a 2024-09-11T05:12:22.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:22.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:22.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 1 -- 172.21.15.138:0/1170476618 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69009cbf0 con 0x7ff690070010 2024-09-11T05:12:22.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 1 -- 172.21.15.138:0/1170476618 >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 msgr2=0x7ff690135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2024-09-11T05:12:22.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 1 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7ff6800098f0 tx=0x7ff68001c470 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-09-11T05:12:22.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.104+0000 7ff68effd640 1 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:22.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68effd640 1 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:22.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68effd640 10 monclient: get_auth_request con 0x7ff690070010 auth_method 0 2024-09-11T05:12:22.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68effd640 0 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T05:12:22.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68effd640 1 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient: ms_handle_reset current mon v2:172.21.15.138:3300/0 2024-09-11T05:12:22.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 1 -- 172.21.15.138:0/1170476618 >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 msgr2=0x7ff690135450 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 1 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.138:3300/0 conn(0x7ff690070010 0x7ff690135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient: _add_conns ranks=[4,2] 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 1 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.190:3300/0 conn(0x7ff678003510 0x7ff678005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient(hunting): picked mon.noname-c con 0x7ff678003510 addr v2:172.21.15.190:3300/0 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 1 --2- 172.21.15.138:0/1170476618 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff678006260 0x7ff678008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient(hunting): picked mon.noname-b con 0x7ff678006260 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 1 -- 172.21.15.138:0/1170476618 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff678008fc0 con 0x7ff678003510 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 1 -- 172.21.15.138:0/1170476618 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff678009170 con 0x7ff678006260 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:22.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68dffb640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.138:3300/0 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68effd640 1 -- 172.21.15.138:0/1170476618 >> v2:172.21.15.190:3300/0 conn(0x7ff678003510 msgr2=0x7ff678005990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68effd640 1 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.190:3300/0 conn(0x7ff678003510 0x7ff678005990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 1 --2- 172.21.15.138:0/1170476618 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff678006260 0x7ff678008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient(hunting): get_auth_request con 0x7ff678006260 auth_method 0 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_done global_id 64197 payload 293 2024-09-11T05:12:22.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 1 -- 172.21.15.138:0/1170476618 >> v2:172.21.15.190:3300/0 conn(0x7ff678003510 msgr2=0x7ff678005990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:12:22.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 1 --2- 172.21.15.138:0/1170476618 >> v2:172.21.15.190:3300/0 conn(0x7ff678003510 0x7ff678005990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient: _finish_hunting 0 2024-09-11T05:12:22.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 1 monclient: found mon.noname-b 2024-09-11T05:12:22.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:22.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:12:22.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.308+0000 7ff68e7fc640 1 -- 172.21.15.138:0/1170476618 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff678009430 con 0x7ff678006260 2024-09-11T05:12:22.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68e7fc640 1 -- 172.21.15.138:0/1170476618 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff678006260 msgr2=0x7ff678008720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:22.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68e7fc640 1 --2- 172.21.15.138:0/1170476618 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff678006260 0x7ff678008720 secure :-1 s=READY pgs=5727 cs=0 l=1 rev1=1 crypto rx=0x7ff684001910 tx=0x7ff680009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6efe63d4c2a05305 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:22.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 1 -- 172.21.15.138:0/1170476618 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684017070 con 0x7ff678006260 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: dump: 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:22.315 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:22.316 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:22.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: _finish_auth 0 2024-09-11T05:12:22.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: _check_auth_tickets 2024-09-11T05:12:22.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:22.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 5 monclient: authenticate success, global_id 64197 2024-09-11T05:12:22.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 1 -- 172.21.15.138:0/1170476618 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6840072d0 con 0x7ff678006260 2024-09-11T05:12:22.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:12:22.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 10 monclient: get_monmap_and_config success 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 4 set_mon_vals no callback set 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 1 -- 172.21.15.138:0/1170476618 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6840016c0 con 0x7ff678006260 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68dffb640 10 monclient: dump: 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:22.317 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 10 monclient: shutdown 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 -- 172.21.15.138:0/1170476618 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff678006260 msgr2=0x7ff678008720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 --2- 172.21.15.138:0/1170476618 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff678006260 0x7ff678008720 secure :-1 s=READY pgs=5727 cs=0 l=1 rev1=1 crypto rx=0x7ff684001910 tx=0x7ff680009240 comp rx=0 tx=0).stop 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff68f7fe640 1 -- 172.21.15.138:0/1170476618 reap_dead start 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 -- 172.21.15.138:0/1170476618 shutdown_connections 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 -- 172.21.15.138:0/1170476618 >> 172.21.15.138:0/1170476618 conn(0x7ff6900ab830 msgr2=0x7ff6900a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 -- 172.21.15.138:0/1170476618 shutdown_connections 2024-09-11T05:12:22.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 -- 172.21.15.138:0/1170476618 wait complete. 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 10 monclient: build_initial_monmap 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 10 monclient: monmap: 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:22.325 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 librados: starting msgr at 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 librados: starting objecter 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 Processor -- start 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 -- start start 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 librados: setting wanted keys 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 1 librados: calling monclient init 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.312+0000 7ff695678640 10 monclient: init 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient: _reopen_session rank -1 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6900a37e0 0x7ff6900a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient(hunting): picked mon.noname-a con 0x7ff6900a37e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:12:22.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6900a4100 0x7ff6901d0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient(hunting): picked mon.noname-b con 0x7ff6900a4100 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff6901d08b0 0x7ff6901d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient(hunting): picked mon.noname-c con 0x7ff6901d08b0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff690143e90 con 0x7ff6900a37e0 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff690144010 con 0x7ff6900a4100 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff690144190 con 0x7ff6901d08b0 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient(hunting): _renew_subs 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 10 monclient(hunting): authenticate will time out at 1225.752716s 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6900a37e0 0x7ff6900a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68e7fc640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6900a4100 msgr2=0x7ff6901d0370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68e7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6900a4100 0x7ff6901d0370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6900a37e0 0x7ff6900a3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:46126/0 (socket says 172.21.15.138:46126) 2024-09-11T05:12:22.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 -- 172.21.15.138:0/569501773 learned_addr learned my addr 172.21.15.138:0/569501773 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient(hunting): get_auth_request con 0x7ff6900a37e0 auth_method 0 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68f7fe640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff6901d08b0 0x7ff6901d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68f7fe640 10 monclient(hunting): get_auth_request con 0x7ff6901d08b0 auth_method 0 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:22.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient(hunting): handle_auth_done global_id 64203 payload 824 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff6901d08b0 msgr2=0x7ff6901d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff6901d08b0 0x7ff6901d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6900a4100 msgr2=0x7ff6901d0370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff6900a4100 0x7ff6901d0370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient: _finish_hunting 0 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 monclient: found mon.noname-a 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 -- 172.21.15.138:0/569501773 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69012f000 con 0x7ff6900a37e0 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6900a37e0 msgr2=0x7ff6900a3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff68effd640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6900a37e0 0x7ff6900a3bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff6800adde0 tx=0x7ff680019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 10 client.?.objecter ms_handle_connect 0x7ff6900a37e0 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800a7070 con 0x7ff6900a37e0 2024-09-11T05:12:22.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 10 monclient: dump: 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:22.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 10 monclient: _finish_auth 0 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 10 monclient: _check_auth_tickets 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff6777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.316+0000 7ff695678640 5 monclient: authenticate success, global_id 64203 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 10 monclient: _renew_subs 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69012fef0 con 0x7ff6900a37e0 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff680004230 con 0x7ff6900a37e0 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800ab840 con 0x7ff6900a37e0 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:12:22.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 10 monclient: dump: 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68cff9640 4 set_mon_vals no callback set 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 10 client.64203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 10 monclient: _renew_subs 2024-09-11T05:12:22.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:12:22.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6900a4540 con 0x7ff6900a37e0 2024-09-11T05:12:22.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 1 librados: init done 2024-09-11T05:12:22.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:22.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f0b0 0x7ff69009f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff6901306f0 con 0x7ff69009f0b0 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f0b0 0x7ff69009f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7ff6800ab9e0 con 0x7ff6900a37e0 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 10 monclient: get_auth_request con 0x7ff69009f0b0 auth_method 0 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:22.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 1 --2- 172.21.15.138:0/569501773 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff67c03ce50 0x7ff67c03f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_done global_id 22535 payload 824 2024-09-11T05:12:22.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 1 -- 172.21.15.138:0/569501773 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6900a2310 con 0x7ff69009f0b0 2024-09-11T05:12:22.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 6213+0+0 (secure 0 0 0) 0x7ff6800ea4b0 con 0x7ff6900a37e0 2024-09-11T05:12:22.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 10 client.64203.objecter ms_dispatch 0x7ff690000ba0 osd_map(209..209 src has 1..209) 2024-09-11T05:12:22.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 3 client.64203.objecter handle_osd_map got epochs [209,209] > 0 2024-09-11T05:12:22.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 3 client.64203.objecter handle_osd_map decoding full epoch 209 2024-09-11T05:12:22.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 20 client.64203.objecter dump_active .. 0 homeless 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68f7fe640 1 --2- 172.21.15.138:0/569501773 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff67c03ce50 0x7ff67c03f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68f7fe640 10 monclient: get_auth_request con 0x7ff67c03ce50 auth_method 0 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 10 client.64203.objecter ms_handle_connect 0x7ff69009f0b0 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff6777fe640 10 client.64203.objecter resend_mon_ops 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f0b0 msgr2=0x7ff69009f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.320+0000 7ff68e7fc640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f0b0 0x7ff69009f490 secure :-1 s=READY pgs=5728 cs=0 l=1 rev1=1 crypto rx=0x7ff6780034a0 tx=0x7ff6840014c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684017030 con 0x7ff69009f0b0 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 10 monclient: dump: 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:22.336 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff684005e20 con 0x7ff69009f0b0 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f0b0 msgr2=0x7ff69009f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f0b0 0x7ff69009f490 secure :-1 s=READY pgs=5728 cs=0 l=1 rev1=1 crypto rx=0x7ff6780034a0 tx=0x7ff6840014c0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f0b0 msgr2=0x7ff69009f490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:22.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f0b0 0x7ff69009f490 secure :-1 s=CLOSED pgs=5728 cs=0 l=1 rev1=1 crypto rx=0x7ff6780034a0 tx=0x7ff6840014c0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff6777fe640 10 client.64203.objecter ms_handle_connect 0x7ff67c03ce50 2024-09-11T05:12:22.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.324+0000 7ff68f7fe640 1 --2- 172.21.15.138:0/569501773 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff67c03ce50 0x7ff67c03f310 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7ff69006daa0 tx=0x7ff6880094e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:22.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff695678640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:12:22.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff695678640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f9d0 0x7ff6901db720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff695678640 10 monclient(hunting): start opening mon connection 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff690135ba0 con 0x7ff69009f9d0 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f9d0 0x7ff6901db720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 10 monclient: get_auth_request con 0x7ff69009f9d0 auth_method 0 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 10 monclient(hunting): handle_auth_done global_id 22538 payload 824 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.328+0000 7ff68e7fc640 1 -- 172.21.15.138:0/569501773 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff690135450 con 0x7ff69009f9d0 2024-09-11T05:12:22.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff68e7fc640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f9d0 msgr2=0x7ff6901db720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:12:22.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 10 client.64203.objecter ms_handle_connect 0x7ff69009f9d0 2024-09-11T05:12:22.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 10 client.64203.objecter resend_mon_ops 2024-09-11T05:12:22.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff68e7fc640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f9d0 0x7ff6901db720 secure :-1 s=READY pgs=5729 cs=0 l=1 rev1=1 crypto rx=0x7ff684015040 tx=0x7ff684007c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6840016c0 con 0x7ff69009f9d0 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 10 monclient: dump: 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:12:22.342 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:12:22.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7ff68400ce40 con 0x7ff69009f9d0 2024-09-11T05:12:22.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:12:22.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:12:22.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f9d0 msgr2=0x7ff6901db720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:22.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f9d0 0x7ff6901db720 secure :-1 s=READY pgs=5729 cs=0 l=1 rev1=1 crypto rx=0x7ff684015040 tx=0x7ff684007c30 comp rx=0 tx=0).stop 2024-09-11T05:12:22.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f9d0 msgr2=0x7ff6901db720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:12:22.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff6777fe640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff69009f9d0 0x7ff6901db720 secure :-1 s=CLOSED pgs=5729 cs=0 l=1 rev1=1 crypto rx=0x7ff684015040 tx=0x7ff684007c30 comp rx=0 tx=0).stop 2024-09-11T05:12:22.343 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 24, 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 301105, 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:22.344 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:22.345 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:22.346 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:12:22.347 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:12:22.348 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.349 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:22.350 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:22.351 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.352 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:12:22.353 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:12:22.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:22.355 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:12:22.356 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:12:22.357 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:12:22.357 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:12:22.357 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:12:22.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:12:22.357 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:12:22.357 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 10 librados: watch_flush enter 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 10 librados: watch_flush exit 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 20 client.64203.objecter shutdown clearing up homeless session... 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 10 client.64203.objecter successfully canceled tick 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff67c03ce50 msgr2=0x7ff67c03f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 --2- 172.21.15.138:0/569501773 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff67c03ce50 0x7ff67c03f310 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7ff69006daa0 tx=0x7ff6880094e0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 10 monclient: shutdown 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6900a37e0 msgr2=0x7ff6900a3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6900a37e0 0x7ff6900a3bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff6800adde0 tx=0x7ff680019210 comp rx=0 tx=0).stop 2024-09-11T05:12:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff68f7fe640 1 -- 172.21.15.138:0/569501773 reap_dead start 2024-09-11T05:12:22.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 shutdown_connections 2024-09-11T05:12:22.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 --2- 172.21.15.138:0/569501773 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff6900a37e0 0x7ff6900a3bc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:12:22.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 >> 172.21.15.138:0/569501773 conn(0x7ff6900ab830 msgr2=0x7ff6900a95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:12:22.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 shutdown_connections 2024-09-11T05:12:22.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 -- 172.21.15.138:0/569501773 wait complete. 2024-09-11T05:12:22.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:12:22.332+0000 7ff695678640 1 librados: shutdown 2024-09-11T05:12:22.360 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:12:22.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:12:24.530 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7010 ms) 2024-09-11T05:12:24.530 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-11T05:12:24.530 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7002 ms) 2024-09-11T05:12:24.530 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-11T05:12:24.530 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7046 ms) 2024-09-11T05:12:24.530 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-11T05:12:24.530 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6957 ms) 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7015 ms) 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7049 ms) 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6961 ms) 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6990 ms) 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7073 ms) 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7047 ms) 2024-09-11T05:12:24.531 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6920 ms) 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6957 ms) 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7055 ms) 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6991 ms) 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6984 ms) 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (21106 ms) 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (126164 ms total) 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] Global test environment tear-down 2024-09-11T05:12:24.532 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (218946 ms total) 2024-09-11T05:12:24.533 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: seed 28187 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (207 ms) 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10333 ms) 2024-09-11T05:12:28.872 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (21 ms) 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10561 ms total) 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6428 ms) 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8054 ms) 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10009 ms) 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: my_snaps [3] 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: my_snaps [4,3] 2024-09-11T05:12:28.873 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting some heads 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: waiting for scrubs... 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: done waiting 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47753 ms) 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9209 ms) 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8035 ms) 2024-09-11T05:12:28.874 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8042 ms) 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7945 ms) 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10665 ms) 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8350 ms) 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10007 ms) 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13102 ms) 2024-09-11T05:12:28.875 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7924 ms) 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7982 ms) 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13038 ms) 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8014 ms) 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-11T05:12:28.876 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (23421 ms) 2024-09-11T05:12:44.141 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-11T05:12:44.141 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-11T05:12:44.141 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: ok, hit_set contains 267:602f83fe:::foo:head 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8568 ms) 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg_num = 32 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 0 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 1 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 2 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 3 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 4 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 5 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 6 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 7 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 8 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 9 ls 1726031564,0 2024-09-11T05:12:44.142 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 10 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 11 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 12 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 13 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 14 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 15 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 16 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 17 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 18 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 19 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 20 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 21 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 22 ls 1726031564,0 2024-09-11T05:12:44.143 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 23 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 24 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 25 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 26 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 27 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 28 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 29 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 30 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 31 ls 1726031564,0 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg_num = 32 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6cac518f:::0:head 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:02547ec2:::1:head 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f905c69b:::2:head 2024-09-11T05:12:44.144 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:cfc208b3:::3:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d83876eb:::4:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b29083e3:::5:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c4fdafeb:::6:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:5c6b0b28:::7:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:bd63b0f1:::8:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e960b815:::9:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:52ea6a34:::10:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:89d3ae78:::11:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:de5d7c5f:::12:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:566253c9:::13:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:62a1935d:::14:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:863748b0:::15:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3958e169:::16:head 2024-09-11T05:12:44.145 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4d4dabf9:::17:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:8391935d:::18:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:28883081:::19:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:69259c59:::20:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4bdb80b7:::21:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a11c5d71:::22:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:271af37b:::23:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:95b121be:::24:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:58d1031b:::25:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0a050783:::26:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c709704c:::27:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:cbe56eaf:::28:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:86b4b162:::29:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:70d89383:::30:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:dd450c7c:::31:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6d5729b1:::32:head 2024-09-11T05:12:44.146 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c388f3fb:::33:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:56cfea31:::34:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9dbc1bf7:::35:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:40b74ccd:::36:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4d5aaf42:::37:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:920f362c:::38:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6cc53222:::39:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9cad833f:::40:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1ea84d41:::41:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c4480ef6:::42:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a694361e:::43:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d1bd33e9:::44:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ddc2cd5d:::45:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:2b782207:::46:head 2024-09-11T05:12:44.147 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7b187fca:::47:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:90ecdf6f:::48:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a5ed95fe:::49:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ea0eaa55:::50:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f33ef17b:::51:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a0d1b2f6:::52:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:60c5229e:::53:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:edcbc575:::54:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:102cf253:::55:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:efb7fb0b:::56:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:50d0a326:::57:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d4dc5daf:::58:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3a130462:::59:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ec87ed71:::60:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d5bc9454:::61:head 2024-09-11T05:12:44.148 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3ddfe313:::62:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7c2816b9:::63:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:47e00e4d:::64:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c6410c18:::65:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b48ed237:::66:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:cd63ad31:::67:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b179e92b:::68:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0d9f741a:::69:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6d3352ae:::70:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c6d5c19e:::71:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:bc4729c3:::72:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:77e930b9:::73:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0abeecfd:::74:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b7c37e15:::75:head 2024-09-11T05:12:44.149 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b6378398:::76:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:02bd68de:::77:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:cc795d2d:::78:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:630d4fea:::79:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e0d29ef5:::80:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:fd6f13d2:::81:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:606461d5:::82:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:eadbdc43:::83:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:8761d0bb:::84:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9ef0186f:::85:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e0d41294:::86:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:961de695:::87:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1423148f:::88:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:633a8fa2:::89:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a8653809:::90:head 2024-09-11T05:12:44.150 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3dac8b33:::91:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:35aad435:::92:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f6dcc343:::93:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:dbbdad87:::94:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1cb48ce0:::95:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:03cd461c:::96:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:17a4ea99:::97:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9993c9a7:::98:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6394211c:::99:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:94c7ae57:::100:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6fdee5bb:::101:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9a477fd1:::102:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:eb850916:::103:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:affc56b9:::104:head 2024-09-11T05:12:44.151 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b42dc814:::105:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f319f8f0:::106:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9a40b9de:::107:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:8b524f28:::108:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e3de589f:::109:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:90f90a5b:::110:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a7b4f1d7:::111:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:af51766e:::112:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b6f90bd1:::113:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e0261208:::114:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c9569ef7:::115:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:61bebe50:::116:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:fe93412b:::117:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d3d38bee:::118:head 2024-09-11T05:12:44.152 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3100ba0c:::119:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d0560ada:::120:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f0ea8b35:::121:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:766f231a:::122:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a07a2582:::123:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:bd7c6b3a:::124:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:fb2ddaff:::125:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4408e1fe:::126:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ee1df7a7:::127:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c3002909:::128:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4f48ffa9:::129:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:edf38733:::130:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c08425c0:::131:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:5f902d98:::132:head 2024-09-11T05:12:44.153 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:41ea2c93:::133:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:813cee13:::134:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0131818d:::135:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:26ba5a85:::136:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:381b8a5a:::137:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:28797e47:::138:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:bfca7f22:::139:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:36807075:::140:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:80b03975:::141:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:5c15709b:::142:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f39ea15e:::143:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ea992956:::144:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:48887b1c:::145:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9f24a9dd:::146:head 2024-09-11T05:12:44.154 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:987f100b:::147:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d2dd3581:::148:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7fed1808:::149:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c80b70e9:::150:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:85ed90f9:::151:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:36428b24:::152:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d044c34a:::153:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7c18bf58:::154:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d1c21232:::155:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a7a3c575:::156:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:87da0633:::157:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d5ac3822:::158:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3f20522d:::159:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6ca26563:::160:head 2024-09-11T05:12:44.155 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:532ce135:::161:head 2024-09-11T05:12:44.156 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c78863e6:::162:head 2024-09-11T05:13:17.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7462ddf6::o.RoundTripAppendPP (3018 ms) 2024-09-11T05:13:17.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-11T05:13:17.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2945 ms) 2024-09-11T05:13:17.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-11T05:13:17.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3008 ms) 2024-09-11T05:13:17.891 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3030 ms) 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: setting pool EIO 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: max_success 98, min_failed 99 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3986 ms) 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3018 ms) 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (130830 ms total) 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3023 ms) 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-11T05:13:17.892 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8971 ms) 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15081 ms) 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2998 ms) 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30073 ms total) 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3028 ms) 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3028 ms total) 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: 2024-09-11T05:13:17.893 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13885 ms) 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7052 ms) 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7008 ms) 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6960 ms) 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (17101 ms) 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7366 ms) 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6548 ms) 2024-09-11T05:13:17.894 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6977 ms) 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7041 ms) 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7141 ms) 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6835 ms) 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7080 ms) 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-11T05:13:17.895 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7389 ms) 2024-09-11T05:13:39.459 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2:163:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:5d165639:::164:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f43765fc:::165:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b4c720e9:::166:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e694b040:::167:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:afa38db2:::168:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:77ba9f53:::169:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:87495034:::170:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7c96bf0e:::171:head 2024-09-11T05:13:39.460 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:dbe346cc:::172:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e943ec24:::173:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f97a9c0c:::174:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6f26e74d:::175:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4f95e106:::176:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0e6f2f8f:::177:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:05db05f1:::178:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:38a78d66:::179:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d095610b:::180:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a1a9d709:::181:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1e5d39db:::182:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f7df4fb9:::183:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:03a7f161:::184:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ba70721e:::185:head 2024-09-11T05:13:39.461 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:28e5662d:::186:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:973d52de:::187:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4303eb1c:::188:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b990b48e:::189:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:29b8165b:::190:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3547f197:::191:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7e260936:::192:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1abec7b1:::193:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:10fdda93:::194:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:15817eea:::195:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:770bab57:::196:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ed9e13e7:::197:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:71471a8f:::198:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:10fb1d02:::199:head 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8987 ms) 2024-09-11T05:13:39.462 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: first is 1726031574 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,1726031579,1726031580,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,1726031579,1726031580,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,1726031579,1726031580,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,1726031579,1726031580,1726031582,1726031583,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,1726031579,1726031580,1726031582,1726031583,0 2024-09-11T05:13:39.463 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031574,1726031576,1726031577,1726031579,1726031580,1726031582,1726031583,0 2024-09-11T05:13:39.464 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726031577,1726031579,1726031580,1726031582,1726031583,1726031585,1726031586,0 2024-09-11T05:13:39.464 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: first now 1726031577, trimmed 2024-09-11T05:13:39.464 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21278 ms) 2024-09-11T05:13:39.464 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-11T05:13:39.464 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: foo0 2024-09-11T05:13:39.464 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-11T05:13:39.464 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13705 ms) 2024-09-11T05:13:39.464 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-11T05:13:52.361 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T05:13:52.361 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T05:13:52.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T05:13:52.365 INFO:tasks.ceph.mon.b:Started 2024-09-11T05:13:52.365 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:13:52.365 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:13:52.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.495+0000 7f52ea5b0640 10 monclient: get_monmap_and_config 2024-09-11T05:13:52.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.495+0000 7f52ea5b0640 10 monclient: build_initial_monmap 2024-09-11T05:13:52.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.495+0000 7f52ea5b0640 10 monclient: monmap: 2024-09-11T05:13:52.498 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:13:52.499 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:52.499 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:13:52.499610+0000 2024-09-11T05:13:52.499 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:13:52.499610+0000 2024-09-11T05:13:52.499 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:13:52.499 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:52.499 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:13:52.499 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:13:52.499 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:13:52.500 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:13:52.500 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:13:52.500 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:52.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.495+0000 7f52ea5b0640 1 Processor -- start 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.495+0000 7f52ea5b0640 1 -- start start 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.495+0000 7f52ea5b0640 10 monclient: init 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 10 monclient: _reopen_session rank -1 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f52e40a0a90 0x7f52e40a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 10 monclient(hunting): picked mon.noname-c con 0x7f52e40a0a90 addr v2:172.21.15.190:3300/0 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52e409dc20 addr v1:172.21.15.138:6789/0 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52e4139be0 addr v1:172.21.15.190:6789/0 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e409efa0 con 0x7f52e409dc20 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e40a2d80 con 0x7f52e4139be0 2024-09-11T05:13:52.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f52e409ecc0 con 0x7f52e40a0a90 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 10 monclient(hunting): _renew_subs 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52ea5b0640 10 monclient(hunting): authenticate will time out at 1315.936367s 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f52e409dc20 0x7f52e40a2670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:54364/0 (socket says 172.21.15.138:54364) 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 1 -- 172.21.15.138:0/170486470 learned_addr learned my addr 172.21.15.138:0/170486470 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 1 --2- 172.21.15.138:0/170486470 >> v2:172.21.15.190:3300/0 conn(0x7f52e40a0a90 0x7f52e40a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): get_auth_request con 0x7f52e40a0a90 auth_method 0 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:52.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:52.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:52.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 1 -- 172.21.15.138:0/170486470 >> v2:172.21.15.190:3300/0 conn(0x7f52e40a0a90 msgr2=0x7f52e40a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:13:52.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 1 -- 172.21.15.138:0/170486470 >> v2:172.21.15.190:3300/0 conn(0x7f52e40a0a90 msgr2=0x7f52e40a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:13:52.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 1 --2- 172.21.15.138:0/170486470 >> v2:172.21.15.190:3300/0 conn(0x7f52e40a0a90 0x7f52e40a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:13:52.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 1 --2- 172.21.15.138:0/170486470 >> v2:172.21.15.190:3300/0 conn(0x7f52e40a0a90 0x7f52e40a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:13:52.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 1 -- 172.21.15.138:0/170486470 >> v1:172.21.15.138:6789/0 conn(0x7f52e409dc20 legacy=0x7f52e40a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:13:52.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e8b4d640 1 -- 172.21.15.138:0/170486470 >> v1:172.21.15.190:6789/0 conn(0x7f52e4139be0 legacy=0x7f52e413bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:13:52.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1806270845 0 0) 0x7f52d8001460 con 0x7f52e409dc20 2024-09-11T05:13:52.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): _negotiate 2024-09-11T05:13:52.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:52.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:52.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): my global_id is 64272 2024-09-11T05:13:52.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52cc003530 con 0x7f52e409dc20 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 693400533 0 0) 0x7f52cc003530 con 0x7f52e409dc20 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 >> v2:172.21.15.190:3300/0 conn(0x7f52e40a0a90 msgr2=0x7f52e40a0e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 --2- 172.21.15.138:0/170486470 >> v2:172.21.15.190:3300/0 conn(0x7f52e40a0a90 0x7f52e40a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 >> v1:172.21.15.190:6789/0 conn(0x7f52e4139be0 legacy=0x7f52e413bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e40a13b0 con 0x7f52e409dc20 2024-09-11T05:13:52.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: _finish_auth 0 2024-09-11T05:13:52.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:13:52.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:13:52.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f52d8001b80 con 0x7f52e409dc20 2024-09-11T05:13:52.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:52.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:13:52.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: dump: 2024-09-11T05:13:52.522 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:52.522 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:52.522 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.522 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.522 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:52.522 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:52.522 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:52.523 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:52.523 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:52.523 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:52.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:13:52.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: _reopen_session rank -1 2024-09-11T05:13:52.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 >> v1:172.21.15.138:6789/0 conn(0x7f52e409dc20 legacy=0x7f52e40a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:52.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:13:52.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52cc003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:52.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): picked mon.a con 0x7f52cc003b30 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:13:52.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52cc004810 0x7f52cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:52.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): picked mon.c con 0x7f52cc004810 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc00bbd0 0x7f52cc00dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): picked mon.b con 0x7f52cc00bbd0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f52e409ecc0 con 0x7f52cc003b30 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc00e5e0 con 0x7f52cc00bbd0 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc00e7e0 con 0x7f52cc004810 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): _renew_subs 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52cc003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): get_auth_request con 0x7f52cc003b30 auth_method 0 2024-09-11T05:13:52.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52cc004810 0x7f52cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 10 monclient(hunting): get_auth_request con 0x7f52cc004810 auth_method 0 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:52.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.499+0000 7f52e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e8b4d640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc00bbd0 0x7f52cc00dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 10 monclient(hunting): handle_auth_done global_id 64272 payload 293 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 1 -- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52cc004810 msgr2=0x7f52cc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52cc004810 0x7f52cc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 1 -- 172.21.15.138:0/170486470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc00bbd0 msgr2=0x7f52cc00dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc00bbd0 0x7f52cc00dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 10 monclient: _finish_hunting 0 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 1 monclient: found mon.a 2024-09-11T05:13:52.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:13:52.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:52.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 1 -- 172.21.15.138:0/170486470 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc003530 con 0x7f52cc003b30 2024-09-11T05:13:52.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 10 monclient: _finish_auth 0 2024-09-11T05:13:52.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 10 monclient: _check_auth_tickets 2024-09-11T05:13:52.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:13:52.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e8b4d640 10 monclient: get_auth_request con 0x7f52cc00bbd0 auth_method 0 2024-09-11T05:13:52.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e8b4d640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52cc00bbd0 0x7f52cc00dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52e8b4d640 1 -- 172.21.15.138:0/170486470 reap_dead start 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52ea5b0640 5 monclient: authenticate success, global_id 64272 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.503+0000 7f52ea5b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 -- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 msgr2=0x7f52cc003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52cc003f50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f52d40072b0 tx=0x7f52d401cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e879b179f9027bdd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d400ac60 con 0x7f52cc003b30 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 10 monclient: dump: 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:52.529 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d400a2f0 con 0x7f52cc003b30 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:13:52.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 1 -- 172.21.15.138:0/170486470 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d40a5a30 con 0x7f52cc003b30 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e2ffd640 10 monclient: dump: 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:52.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient: get_monmap_and_config success 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 4 set_mon_vals no callback set 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient: shutdown 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 msgr2=0x7f52cc003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52cc003f50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f52d40072b0 tx=0x7f52d401cd00 comp rx=0 tx=0).stop 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- 172.21.15.138:0/170486470 shutdown_connections 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 --2- 172.21.15.138:0/170486470 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52cc003f50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- 172.21.15.138:0/170486470 >> 172.21.15.138:0/170486470 conn(0x7f52e4063640 msgr2=0x7f52e4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- 172.21.15.138:0/170486470 shutdown_connections 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- 172.21.15.138:0/170486470 wait complete. 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient: build_initial_monmap 2024-09-11T05:13:52.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient: monmap: 2024-09-11T05:13:52.533 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:13:52.533 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:13:52.533 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:13:52.533 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:13:52.533 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:13:52.533 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 librados: starting msgr at 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 librados: starting objecter 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 Processor -- start 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- start start 2024-09-11T05:13:52.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 librados: setting wanted keys 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 librados: calling monclient init 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient: init 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient: _reopen_session rank -1 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52e41d1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient(hunting): picked mon.noname-a con 0x7f52cc003b30 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e41339d0 0x7f52e4133e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient(hunting): picked mon.noname-b con 0x7f52e41339d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52e4124650 0x7f52e4124ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient(hunting): picked mon.noname-c con 0x7f52e4124650 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f52e4134c50 con 0x7f52cc003b30 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:52.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f52e4134ad0 con 0x7f52e41339d0 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f52e4134dd0 con 0x7f52e4124650 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient(hunting): _renew_subs 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52ea5b0640 10 monclient(hunting): authenticate will time out at 1315.946505s 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52e41d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52e4124650 0x7f52e4124ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52e41d1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45690/0 (socket says 172.21.15.138:45690) 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 -- 172.21.15.138:0/2826555035 learned_addr learned my addr 172.21.15.138:0/2826555035 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient(hunting): get_auth_request con 0x7f52cc003b30 auth_method 0 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:52.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 10 monclient(hunting): get_auth_request con 0x7f52e4124650 auth_method 0 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:52.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e37fe640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e41339d0 0x7f52e4133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e37fe640 10 monclient(hunting): get_auth_request con 0x7f52e41339d0 auth_method 0 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient(hunting): handle_auth_done global_id 64275 payload 824 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 -- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52e4124650 msgr2=0x7f52e4124ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52e4124650 0x7f52e4124ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 -- 172.21.15.138:0/2826555035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e41339d0 msgr2=0x7f52e4133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e41339d0 0x7f52e4133e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient: _finish_hunting 0 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 monclient: found mon.noname-a 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e3fff640 1 -- 172.21.15.138:0/2826555035 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e4128420 con 0x7f52cc003b30 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.507+0000 7f52e8b4d640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52e4124650 0x7f52e4124ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e3fff640 1 -- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 msgr2=0x7f52e41d1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e3fff640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52e41d1070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f52d401cb30 tx=0x7f52cc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 client.?.objecter ms_handle_connect 0x7f52cc003b30 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 1 -- 172.21.15.138:0/2826555035 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d40ba030 con 0x7f52cc003b30 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:13:52.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: dump: 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:52.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: _finish_auth 0 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 5 monclient: authenticate success, global_id 64275 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 10 monclient: _renew_subs 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 1 -- 172.21.15.138:0/2826555035 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d40b6440 con 0x7f52cc003b30 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e40a68f0 con 0x7f52cc003b30 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 1 -- 172.21.15.138:0/2826555035 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d40b6830 con 0x7f52cc003b30 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 monclient: dump: 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:52.541 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:52.542 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.542 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:52.542 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:52.542 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:52.542 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:52.542 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:52.542 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:52.542 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 10 client.64275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 10 monclient: _renew_subs 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e4134370 con 0x7f52cc003b30 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e1ffb640 4 set_mon_vals no callback set 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 1 librados: init done 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e4135160 con 0x7f52cc003b30 2024-09-11T05:13:52.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 1 -- 172.21.15.138:0/2826555035 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f52d40b6a10 con 0x7f52cc003b30 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 1 --2- 172.21.15.138:0/2826555035 >> v2:172.21.15.190:6800/1562782189 conn(0x7f52d003cea0 0x7f52d003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 1 -- 172.21.15.138:0/2826555035 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(300..300 src has 1..300) ==== 6231+0+0 (secure 0 0 0) 0x7f52d40ea700 con 0x7f52cc003b30 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 10 client.64275.objecter ms_dispatch 0x7f52e4000ba0 osd_map(300..300 src has 1..300) 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 3 client.64275.objecter handle_osd_map got epochs [300,300] > 0 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 3 client.64275.objecter handle_osd_map decoding full epoch 300 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e0ff9640 20 client.64275.objecter dump_active .. 0 homeless 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e37fe640 1 --2- 172.21.15.138:0/2826555035 >> v2:172.21.15.190:6800/1562782189 conn(0x7f52d003cea0 0x7f52d003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.511+0000 7f52e37fe640 10 monclient: get_auth_request con 0x7f52d003cea0 auth_method 0 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.515+0000 7f52e37fe640 1 --2- 172.21.15.138:0/2826555035 >> v2:172.21.15.190:6800/1562782189 conn(0x7f52d003cea0 0x7f52d003f360 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f52d80010f0 tx=0x7f52d800a040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:52.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.515+0000 7f52e0ff9640 10 client.64275.objecter ms_handle_connect 0x7f52d003cea0 2024-09-11T05:13:52.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.515+0000 7f52e0ff9640 1 -- 172.21.15.138:0/2826555035 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f52d40c4070 con 0x7f52cc003b30 2024-09-11T05:13:52.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.515+0000 7f52e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:52.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.515+0000 7f52e0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:13:52.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.751+0000 7f52ea5b0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:13:52.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.751+0000 7f52ea5b0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:13:52.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.751+0000 7f52ea5b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:52.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.751+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f52e409e110 con 0x7f52cc003b30 2024-09-11T05:13:52.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.751+0000 7f52e0ff9640 1 -- 172.21.15.138:0/2826555035 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f52d4016a10 con 0x7f52cc003b30 2024-09-11T05:13:52.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.751+0000 7f52e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:13:52.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.751+0000 7f52e0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:13:52.754 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:13:52.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 10 librados: watch_flush enter 2024-09-11T05:13:52.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 10 librados: watch_flush exit 2024-09-11T05:13:52.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 20 client.64275.objecter shutdown clearing up homeless session... 2024-09-11T05:13:52.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 10 client.64275.objecter successfully canceled tick 2024-09-11T05:13:52.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 >> v2:172.21.15.190:6800/1562782189 conn(0x7f52d003cea0 msgr2=0x7f52d003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:52.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 --2- 172.21.15.138:0/2826555035 >> v2:172.21.15.190:6800/1562782189 conn(0x7f52d003cea0 0x7f52d003f360 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f52d80010f0 tx=0x7f52d800a040 comp rx=0 tx=0).stop 2024-09-11T05:13:52.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 10 monclient: shutdown 2024-09-11T05:13:52.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 msgr2=0x7f52e41d1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52e41d1070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f52d401cb30 tx=0x7f52cc005840 comp rx=0 tx=0).stop 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 shutdown_connections 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 --2- 172.21.15.138:0/2826555035 >> v2:172.21.15.190:6800/1562782189 conn(0x7f52d003cea0 0x7f52d003f360 unknown :-1 s=CLOSED pgs=3393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52e4124650 0x7f52e4124ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52e41339d0 0x7f52e4133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 --2- 172.21.15.138:0/2826555035 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52cc003b30 0x7f52e41d1070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 >> 172.21.15.138:0/2826555035 conn(0x7f52e4063640 msgr2=0x7f52e40a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 shutdown_connections 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 -- 172.21.15.138:0/2826555035 wait complete. 2024-09-11T05:13:52.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:52.755+0000 7f52ea5b0640 1 librados: shutdown 2024-09-11T05:13:55.774 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:13:55.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient: get_monmap_and_config 2024-09-11T05:13:55.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient: build_initial_monmap 2024-09-11T05:13:55.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient: monmap: 2024-09-11T05:13:55.901 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:13:55.901 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:13:55.902960+0000 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:13:55.902960+0000 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:13:55.902 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 1 Processor -- start 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 1 -- start start 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient: init 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient: _reopen_session rank -1 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64d406e0e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f64d409eb30 0x7f64d413a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient(hunting): picked mon.noname-a con 0x7f64d409eb30 addr v2:172.21.15.138:3300/0 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409ef40 0x7f64d413ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:55.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient(hunting): picked mon.noname-b con 0x7f64d409ef40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64d406e4c0 con 0x7f64d406e0e0 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f64d409f820 con 0x7f64d409eb30 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f64d409f540 con 0x7f64d409ef40 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient(hunting): _renew_subs 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64dadee640 10 monclient(hunting): authenticate will time out at 1319.339139s 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f64d409eb30 0x7f64d413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409ef40 0x7f64d413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409ef40 0x7f64d413ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:33728/0 (socket says 172.21.15.138:33728) 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 1 -- 172.21.15.138:0/3379290217 learned_addr learned my addr 172.21.15.138:0/3379290217 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f64d409eb30 0x7f64d413a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45706/0 (socket says 172.21.15.138:45706) 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient(hunting): get_auth_request con 0x7f64d409ef40 auth_method 0 2024-09-11T05:13:55.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 10 monclient(hunting): get_auth_request con 0x7f64d409eb30 auth_method 0 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient(hunting): handle_auth_done global_id 22622 payload 274 2024-09-11T05:13:55.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 1 -- 172.21.15.138:0/3379290217 >> v2:172.21.15.138:3300/0 conn(0x7f64d409eb30 msgr2=0x7f64d413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:13:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 1 --2- 172.21.15.138:0/3379290217 >> v2:172.21.15.138:3300/0 conn(0x7f64d409eb30 0x7f64d413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 1 -- 172.21.15.138:0/3379290217 >> v1:172.21.15.190:6789/0 conn(0x7f64d406e0e0 legacy=0x7f64d409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:13:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient: _finish_hunting 0 2024-09-11T05:13:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 1 monclient: found mon.noname-b 2024-09-11T05:13:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:13:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:13:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d938b640 1 -- 172.21.15.138:0/3379290217 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d409fb00 con 0x7f64d409ef40 2024-09-11T05:13:55.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.898+0000 7f64d3fff640 1 --2- 172.21.15.138:0/3379290217 >> v2:172.21.15.138:3300/0 conn(0x7f64d409eb30 0x7f64d413a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d938b640 1 -- 172.21.15.138:0/3379290217 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409ef40 msgr2=0x7f64d413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d938b640 1 --2- 172.21.15.138:0/3379290217 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409ef40 0x7f64d413ce80 secure :-1 s=READY pgs=5762 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009e50 tx=0x7f64cc01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=406482a20be7f17a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 1 -- 172.21.15.138:0/3379290217 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0a5670 con 0x7f64d409ef40 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: dump: 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:55.912 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: _finish_auth 0 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: _check_auth_tickets 2024-09-11T05:13:55.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 1 -- 172.21.15.138:0/3379290217 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64cc0a5810 con 0x7f64d409ef40 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 5 monclient: authenticate success, global_id 22622 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 1 -- 172.21.15.138:0/3379290217 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0a5be0 con 0x7f64d409ef40 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d37fe640 10 monclient: dump: 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:55.914 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient: get_monmap_and_config success 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 4 set_mon_vals no callback set 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient: shutdown 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- 172.21.15.138:0/3379290217 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409ef40 msgr2=0x7f64d413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 --2- 172.21.15.138:0/3379290217 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409ef40 0x7f64d413ce80 secure :-1 s=READY pgs=5762 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009e50 tx=0x7f64cc01c8f0 comp rx=0 tx=0).stop 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- 172.21.15.138:0/3379290217 shutdown_connections 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 --2- 172.21.15.138:0/3379290217 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409ef40 0x7f64d413ce80 unknown :-1 s=CLOSED pgs=5762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 --2- 172.21.15.138:0/3379290217 >> v2:172.21.15.138:3300/0 conn(0x7f64d409eb30 0x7f64d413a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- 172.21.15.138:0/3379290217 >> 172.21.15.138:0/3379290217 conn(0x7f64d4063640 msgr2=0x7f64d4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:13:55.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- 172.21.15.138:0/3379290217 shutdown_connections 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- 172.21.15.138:0/3379290217 wait complete. 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient: build_initial_monmap 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient: monmap: 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:55.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 librados: starting msgr at 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 librados: starting objecter 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 Processor -- start 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- start start 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 librados: setting wanted keys 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 librados: calling monclient init 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient: init 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient: _reopen_session rank -1 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d406e0e0 0x7f64d411da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient(hunting): picked mon.noname-a con 0x7f64d406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409eb30 0x7f64d411f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient(hunting): picked mon.noname-c con 0x7f64d409eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64d409ef40 0x7f64d4135820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:55.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient(hunting): picked mon.noname-b con 0x7f64d409ef40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f64d413dd50 con 0x7f64d406e0e0 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64d413dbd0 con 0x7f64d409ef40 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f64d413ded0 con 0x7f64d409eb30 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient(hunting): _renew_subs 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64dadee640 10 monclient(hunting): authenticate will time out at 1319.346547s 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409eb30 0x7f64d411f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d8b8a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d406e0e0 0x7f64d411da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409eb30 0x7f64d411f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:33734/0 (socket says 172.21.15.138:33734) 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 -- 172.21.15.138:0/2018619134 learned_addr learned my addr 172.21.15.138:0/2018619134 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:13:55.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient(hunting): get_auth_request con 0x7f64d409eb30 auth_method 0 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d8b8a640 10 monclient(hunting): get_auth_request con 0x7f64d406e0e0 auth_method 0 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d8b8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d8b8a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d8b8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d8b8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d8b8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d8b8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient(hunting): handle_auth_done global_id 22625 payload 824 2024-09-11T05:13:55.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d938b640 1 --2- 172.21.15.138:0/2018619134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64d409ef40 0x7f64d4135820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 -- 172.21.15.138:0/2018619134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64d409ef40 msgr2=0x7f64d4135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 --2- 172.21.15.138:0/2018619134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64d409ef40 0x7f64d4135820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 -- 172.21.15.138:0/2018619134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d406e0e0 msgr2=0x7f64d411da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 --2- 172.21.15.138:0/2018619134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d406e0e0 0x7f64d411da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:55.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient: _finish_hunting 0 2024-09-11T05:13:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 monclient: found mon.noname-c 2024-09-11T05:13:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:13:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:13:55.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.906+0000 7f64d3fff640 1 -- 172.21.15.138:0/2018619134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d413ca00 con 0x7f64d409eb30 2024-09-11T05:13:55.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d3fff640 1 -- 172.21.15.138:0/2018619134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409eb30 msgr2=0x7f64d411f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d3fff640 1 --2- 172.21.15.138:0/2018619134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409eb30 0x7f64d411f9c0 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 crypto rx=0x7f64c800ebe0 tx=0x7f64c800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 client.?.objecter ms_handle_connect 0x7f64d409eb30 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 1 -- 172.21.15.138:0/2018619134 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8017070 con 0x7f64d409eb30 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: dump: 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:55.924 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: _finish_auth 0 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: _check_auth_tickets 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 1 -- 172.21.15.138:0/2018619134 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c800ce40 con 0x7f64d409eb30 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 5 monclient: authenticate success, global_id 22625 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 10 monclient: _renew_subs 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d4120920 con 0x7f64d409eb30 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:13:55.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 1 -- 172.21.15.138:0/2018619134 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8013690 con 0x7f64d409eb30 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 monclient: dump: 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:55.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d2ffd640 4 set_mon_vals no callback set 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 10 client.22625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 10 monclient: _renew_subs 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d4135f30 con 0x7f64d409eb30 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 1 librados: init done 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 1 -- 172.21.15.138:0/2018619134 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f64c8024060 con 0x7f64d409eb30 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d409e420 con 0x7f64d409eb30 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 1 --2- 172.21.15.138:0/2018619134 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64c003ced0 0x7f64c003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 1 -- 172.21.15.138:0/2018619134 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(303..303 src has 1..303) ==== 5408+0+0 (secure 0 0 0) 0x7f64c804f9e0 con 0x7f64d409eb30 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 10 client.22625.objecter ms_dispatch 0x7f64d4000ba0 osd_map(303..303 src has 1..303) 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 3 client.22625.objecter handle_osd_map got epochs [303,303] > 0 2024-09-11T05:13:55.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 3 client.22625.objecter handle_osd_map decoding full epoch 303 2024-09-11T05:13:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d17fa640 20 client.22625.objecter dump_active .. 0 homeless 2024-09-11T05:13:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d8b8a640 1 --2- 172.21.15.138:0/2018619134 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64c003ced0 0x7f64c003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.910+0000 7f64d8b8a640 10 monclient: get_auth_request con 0x7f64c003ced0 auth_method 0 2024-09-11T05:13:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.914+0000 7f64d17fa640 10 client.22625.objecter ms_handle_connect 0x7f64c003ced0 2024-09-11T05:13:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.914+0000 7f64d8b8a640 1 --2- 172.21.15.138:0/2018619134 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64c003ced0 0x7f64c003f390 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7f64c40098e0 tx=0x7f64c4009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.914+0000 7f64d17fa640 1 -- 172.21.15.138:0/2018619134 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f64d413ca00 con 0x7f64d409eb30 2024-09-11T05:13:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.914+0000 7f64d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:55.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:55.914+0000 7f64d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:13:56.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.150+0000 7f64dadee640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:13:56.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.150+0000 7f64dadee640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:13:56.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.150+0000 7f64dadee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:13:56.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.150+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f64d409e0e0 con 0x7f64d409eb30 2024-09-11T05:13:56.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64d17fa640 1 -- 172.21.15.138:0/2018619134 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f64d409e0e0 con 0x7f64d409eb30 2024-09-11T05:13:56.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:13:56.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:13:56.156 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 10 librados: watch_flush enter 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 10 librados: watch_flush exit 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 20 client.22625.objecter shutdown clearing up homeless session... 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 10 client.22625.objecter successfully canceled tick 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64c003ced0 msgr2=0x7f64c003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 --2- 172.21.15.138:0/2018619134 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64c003ced0 0x7f64c003f390 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7f64c40098e0 tx=0x7f64c4009060 comp rx=0 tx=0).stop 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 10 monclient: shutdown 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409eb30 msgr2=0x7f64d411f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 --2- 172.21.15.138:0/2018619134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409eb30 0x7f64d411f9c0 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 crypto rx=0x7f64c800ebe0 tx=0x7f64c800c520 comp rx=0 tx=0).stop 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 shutdown_connections 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 --2- 172.21.15.138:0/2018619134 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64c003ced0 0x7f64c003f390 unknown :-1 s=CLOSED pgs=3395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 --2- 172.21.15.138:0/2018619134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64d409ef40 0x7f64d4135820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:56.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 --2- 172.21.15.138:0/2018619134 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f64d409eb30 0x7f64d411f9c0 unknown :-1 s=CLOSED pgs=5763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:56.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 --2- 172.21.15.138:0/2018619134 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64d406e0e0 0x7f64d411da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:56.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 >> 172.21.15.138:0/2018619134 conn(0x7f64d4063640 msgr2=0x7f64d4136910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:13:56.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 shutdown_connections 2024-09-11T05:13:56.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 -- 172.21.15.138:0/2018619134 wait complete. 2024-09-11T05:13:56.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:56.154+0000 7f64dadee640 1 librados: shutdown 2024-09-11T05:13:59.174 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:13:59.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient: get_monmap_and_config 2024-09-11T05:13:59.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient: build_initial_monmap 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient: monmap: 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:13:59.303201+0000 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:13:59.303201+0000 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:13:59.302 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:59.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 1 Processor -- start 2024-09-11T05:13:59.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 1 -- start start 2024-09-11T05:13:59.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient: init 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient: _reopen_session rank -1 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd8740a0a90 0x7fd8740a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient(hunting): picked mon.noname-a con 0x7fd8740a0a90 addr v2:172.21.15.138:3300/0 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd87409dc20 addr v1:172.21.15.190:6789/0 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd874139be0 0x7fd87413bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient(hunting): picked mon.noname-c con 0x7fd874139be0 addr v2:172.21.15.190:3300/0 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd87409efa0 con 0x7fd87409dc20 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fd8740a2d80 con 0x7fd8740a0a90 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd87409ecc0 con 0x7fd874139be0 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient(hunting): _renew_subs 2024-09-11T05:13:59.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.298+0000 7fd878a10640 10 monclient(hunting): authenticate will time out at 1322.739865s 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd8740a0a90 0x7fd8740a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd8740a0a90 0x7fd8740a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43878/0 (socket says 172.21.15.138:43878) 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 1 -- 172.21.15.138:0/724452458 learned_addr learned my addr 172.21.15.138:0/724452458 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient(hunting): get_auth_request con 0x7fd8740a0a90 auth_method 0 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd872d9d640 1 --2- 172.21.15.138:0/724452458 >> v2:172.21.15.190:3300/0 conn(0x7fd874139be0 0x7fd87413bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient(hunting): handle_auth_done global_id 64290 payload 274 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 1 -- 172.21.15.138:0/724452458 >> v2:172.21.15.190:3300/0 conn(0x7fd874139be0 msgr2=0x7fd87413bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:13:59.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 1 --2- 172.21.15.138:0/724452458 >> v2:172.21.15.190:3300/0 conn(0x7fd874139be0 0x7fd87413bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 1 -- 172.21.15.138:0/724452458 >> v1:172.21.15.190:6789/0 conn(0x7fd87409dc20 legacy=0x7fd8740a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:13:59.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient: _finish_hunting 0 2024-09-11T05:13:59.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 1 monclient: found mon.noname-a 2024-09-11T05:13:59.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:13:59.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:13:59.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd87259c640 1 -- 172.21.15.138:0/724452458 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8740a13b0 con 0x7fd8740a0a90 2024-09-11T05:13:59.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd872d9d640 10 monclient: get_auth_request con 0x7fd874139be0 auth_method 0 2024-09-11T05:13:59.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.302+0000 7fd872d9d640 1 --2- 172.21.15.138:0/724452458 >> v2:172.21.15.190:3300/0 conn(0x7fd874139be0 0x7fd87413bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:13:59.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 1 -- 172.21.15.138:0/724452458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8740a0a90 msgr2=0x7fd8740a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:13:59.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 1 --2- 172.21.15.138:0/724452458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8740a0a90 0x7fd8740a0e70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd8740a4d20 tx=0x7fd86401c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80fb13ed43c8595a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:59.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 1 -- 172.21.15.138:0/724452458 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8640aa480 con 0x7fd8740a0a90 2024-09-11T05:13:59.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:59.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:13:59.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: dump: 2024-09-11T05:13:59.312 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:59.312 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: _finish_auth 0 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: _check_auth_tickets 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 5 monclient: authenticate success, global_id 64290 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 1 -- 172.21.15.138:0/724452458 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8640af070 con 0x7fd8740a0a90 2024-09-11T05:13:59.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient: get_monmap_and_config success 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 4 set_mon_vals no callback set 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 1 -- 172.21.15.138:0/724452458 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8640aa840 con 0x7fd8740a0a90 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87159a640 10 monclient: dump: 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:59.314 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient: shutdown 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- 172.21.15.138:0/724452458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8740a0a90 msgr2=0x7fd8740a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 --2- 172.21.15.138:0/724452458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8740a0a90 0x7fd8740a0e70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd8740a4d20 tx=0x7fd86401c6d0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- 172.21.15.138:0/724452458 shutdown_connections 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 --2- 172.21.15.138:0/724452458 >> v2:172.21.15.190:3300/0 conn(0x7fd874139be0 0x7fd87413bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 --2- 172.21.15.138:0/724452458 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd8740a0a90 0x7fd8740a0e70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- 172.21.15.138:0/724452458 >> 172.21.15.138:0/724452458 conn(0x7fd874063640 msgr2=0x7fd874063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- 172.21.15.138:0/724452458 shutdown_connections 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- 172.21.15.138:0/724452458 wait complete. 2024-09-11T05:13:59.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient: build_initial_monmap 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient: monmap: 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 librados: starting msgr at 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 librados: starting objecter 2024-09-11T05:13:59.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 Processor -- start 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- start start 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 librados: setting wanted keys 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 librados: calling monclient init 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient: init 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient: _reopen_session rank -1 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd87409dc20 0x7fd874073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient(hunting): picked mon.noname-c con 0x7fd87409dc20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8740a0a90 0x7fd87412fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient(hunting): picked mon.noname-b con 0x7fd8740a0a90 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd874139be0 0x7fd8741cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient(hunting): picked mon.noname-a con 0x7fd874139be0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:59.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd87413d170 con 0x7fd874139be0 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd87413cff0 con 0x7fd8740a0a90 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient(hunting): start opening mon connection 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd87413d2f0 con 0x7fd87409dc20 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient(hunting): _renew_subs 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd878a10640 10 monclient(hunting): authenticate will time out at 1322.747335s 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd874139be0 0x7fd8741cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd87409dc20 0x7fd874073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:59.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd874139be0 0x7fd8741cd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43898/0 (socket says 172.21.15.138:43898) 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 -- 172.21.15.138:0/3024605830 learned_addr learned my addr 172.21.15.138:0/3024605830 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient(hunting): get_auth_request con 0x7fd874139be0 auth_method 0 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 10 monclient(hunting): get_auth_request con 0x7fd87409dc20 auth_method 0 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:59.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd87259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd871d9b640 1 --2- 172.21.15.138:0/3024605830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8740a0a90 0x7fd87412fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd871d9b640 10 monclient(hunting): get_auth_request con 0x7fd8740a0a90 auth_method 0 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd871d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd871d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd871d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient(hunting): handle_auth_done global_id 64293 payload 824 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 -- 172.21.15.138:0/3024605830 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd87409dc20 msgr2=0x7fd874073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 --2- 172.21.15.138:0/3024605830 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd87409dc20 0x7fd874073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 -- 172.21.15.138:0/3024605830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8740a0a90 msgr2=0x7fd87412fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 --2- 172.21.15.138:0/3024605830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8740a0a90 0x7fd87412fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:13:59.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 monclient: found mon.noname-a 2024-09-11T05:13:59.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:13:59.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:13:59.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.306+0000 7fd872d9d640 1 -- 172.21.15.138:0/3024605830 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87413b490 con 0x7fd874139be0 2024-09-11T05:13:59.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd872d9d640 1 -- 172.21.15.138:0/3024605830 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd874139be0 msgr2=0x7fd8741cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:13:59.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd872d9d640 1 --2- 172.21.15.138:0/3024605830 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd874139be0 0x7fd8741cd6f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00c820 tx=0x7fd86c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:59.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 client.?.objecter ms_handle_connect 0x7fd874139be0 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 1 -- 172.21.15.138:0/3024605830 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c015070 con 0x7fd874139be0 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: dump: 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:59.327 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: _finish_auth 0 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 1 -- 172.21.15.138:0/3024605830 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86c0079e0 con 0x7fd874139be0 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 1 -- 172.21.15.138:0/3024605830 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c007dd0 con 0x7fd874139be0 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 monclient: dump: 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:13:59.328 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 5 monclient: authenticate success, global_id 64293 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd870d99640 4 set_mon_vals no callback set 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 10 monclient: _renew_subs 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8741d10d0 con 0x7fd874139be0 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 10 client.64293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:13:59.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 10 monclient: _renew_subs 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8740a4e20 con 0x7fd874139be0 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 1 librados: init done 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8740a0e70 con 0x7fd874139be0 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 1 -- 172.21.15.138:0/3024605830 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fd86c0113f0 con 0x7fd874139be0 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 1 --2- 172.21.15.138:0/3024605830 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd85403ced0 0x7fd85403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 1 -- 172.21.15.138:0/3024605830 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(306..306 src has 1..306) ==== 6255+0+0 (secure 0 0 0) 0x7fd86c050110 con 0x7fd874139be0 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 10 client.64293.objecter ms_dispatch 0x7fd874000ba0 osd_map(306..306 src has 1..306) 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 3 client.64293.objecter handle_osd_map got epochs [306,306] > 0 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 3 client.64293.objecter handle_osd_map decoding full epoch 306 2024-09-11T05:13:59.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd862ffd640 20 client.64293.objecter dump_active .. 0 homeless 2024-09-11T05:13:59.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.310+0000 7fd87259c640 1 --2- 172.21.15.138:0/3024605830 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd85403ced0 0x7fd85403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:13:59.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.314+0000 7fd87259c640 10 monclient: get_auth_request con 0x7fd85403ced0 auth_method 0 2024-09-11T05:13:59.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.314+0000 7fd862ffd640 10 client.64293.objecter ms_handle_connect 0x7fd85403ced0 2024-09-11T05:13:59.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.314+0000 7fd87259c640 1 --2- 172.21.15.138:0/3024605830 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd85403ced0 0x7fd85403f390 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fd87406a140 tx=0x7fd8640b5040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:13:59.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.318+0000 7fd862ffd640 1 -- 172.21.15.138:0/3024605830 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fd8741d10d0 con 0x7fd874139be0 2024-09-11T05:13:59.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.318+0000 7fd862ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:13:59.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.318+0000 7fd862ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:13:59.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.558+0000 7fd878a10640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:13:59.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.558+0000 7fd878a10640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:13:59.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.558+0000 7fd878a10640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:13:59.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.558+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd8741397e0 con 0x7fd874139be0 2024-09-11T05:13:59.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd862ffd640 1 -- 172.21.15.138:0/3024605830 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fd86c018c80 con 0x7fd874139be0 2024-09-11T05:13:59.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd862ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:13:59.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd862ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:13:59.565 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 10 librados: watch_flush enter 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 10 librados: watch_flush exit 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 20 client.64293.objecter shutdown clearing up homeless session... 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 10 client.64293.objecter successfully canceled tick 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd85403ced0 msgr2=0x7fd85403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 --2- 172.21.15.138:0/3024605830 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd85403ced0 0x7fd85403f390 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fd87406a140 tx=0x7fd8640b5040 comp rx=0 tx=0).stop 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 10 monclient: shutdown 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd874139be0 msgr2=0x7fd8741cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:13:59.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 --2- 172.21.15.138:0/3024605830 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd874139be0 0x7fd8741cd6f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00c820 tx=0x7fd86c00cce0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 shutdown_connections 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 --2- 172.21.15.138:0/3024605830 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd85403ced0 0x7fd85403f390 unknown :-1 s=CLOSED pgs=3396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 --2- 172.21.15.138:0/3024605830 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd874139be0 0x7fd8741cd6f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 --2- 172.21.15.138:0/3024605830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8740a0a90 0x7fd87412fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 --2- 172.21.15.138:0/3024605830 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd87409dc20 0x7fd874073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 >> 172.21.15.138:0/3024605830 conn(0x7fd874063640 msgr2=0x7fd87413d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 shutdown_connections 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 -- 172.21.15.138:0/3024605830 wait complete. 2024-09-11T05:13:59.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:13:59.562+0000 7fd878a10640 1 librados: shutdown 2024-09-11T05:14:02.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ 2024-09-11T05:14:02.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2568 ms) 2024-09-11T05:14:02.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-11T05:14:02.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6993 ms) 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6994 ms) 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6980 ms) 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7045 ms) 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6976 ms) 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7003 ms) 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (152942 ms total) 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-11T05:14:02.457 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (316873 ms total) 2024-09-11T05:14:02.458 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-11T05:14:02.583 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:14:02.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient: get_monmap_and_config 2024-09-11T05:14:02.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient: build_initial_monmap 2024-09-11T05:14:02.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient: monmap: 2024-09-11T05:14:02.713 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:02.713 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:02.713 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:02.714644+0000 2024-09-11T05:14:02.713 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:02.714644+0000 2024-09-11T05:14:02.713 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:02.714 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:02.714 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:02.714 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:02.714 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:02.714 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:02.714 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:02.714 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:02.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 1 Processor -- start 2024-09-11T05:14:02.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 1 -- start start 2024-09-11T05:14:02.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient: init 2024-09-11T05:14:02.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:02.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce04136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient(hunting): picked mon.noname-b con 0x7fce0406e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fce04136170 0x7fce04136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient(hunting): picked mon.noname-c con 0x7fce04136170 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fce040a1920 0x7fce040a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient(hunting): picked mon.noname-a con 0x7fce040a1920 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fce0406e4c0 con 0x7fce040a1920 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fce040a3270 con 0x7fce04136170 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fce040a0af0 con 0x7fce0406e0e0 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:02.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce09151640 10 monclient(hunting): authenticate will time out at 1326.150786s 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fce040a1920 0x7fce040a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce04136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fce040a1920 0x7fce040a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43906/0 (socket says 172.21.15.138:43906) 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce04136720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:53962/0 (socket says 172.21.15.138:53962) 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 1 -- 172.21.15.138:0/1282362454 learned_addr learned my addr 172.21.15.138:0/1282362454 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 10 monclient(hunting): get_auth_request con 0x7fce040a1920 auth_method 0 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient(hunting): get_auth_request con 0x7fce0406e0e0 auth_method 0 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:02.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient(hunting): handle_auth_done global_id 22631 payload 274 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0259c640 1 --2- 172.21.15.138:0/1282362454 >> v2:172.21.15.190:3300/0 conn(0x7fce04136170 0x7fce04136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 1 -- 172.21.15.138:0/1282362454 >> v2:172.21.15.190:3300/0 conn(0x7fce04136170 msgr2=0x7fce04136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 1 --2- 172.21.15.138:0/1282362454 >> v2:172.21.15.190:3300/0 conn(0x7fce04136170 0x7fce04136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 1 -- 172.21.15.138:0/1282362454 >> v2:172.21.15.138:3300/0 conn(0x7fce040a1920 msgr2=0x7fce040a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 1 --2- 172.21.15.138:0/1282362454 >> v2:172.21.15.138:3300/0 conn(0x7fce040a1920 0x7fce040a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:14:02.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 1 monclient: found mon.noname-b 2024-09-11T05:14:02.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:02.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:14:02.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce02d9d640 1 -- 172.21.15.138:0/1282362454 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce040a2f90 con 0x7fce0406e0e0 2024-09-11T05:14:02.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.710+0000 7fce0359e640 1 --2- 172.21.15.138:0/1282362454 >> v2:172.21.15.138:3300/0 conn(0x7fce040a1920 0x7fce040a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:02.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce02d9d640 1 -- 172.21.15.138:0/1282362454 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 msgr2=0x7fce04136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:02.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce02d9d640 1 --2- 172.21.15.138:0/1282362454 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce04136720 secure :-1 s=READY pgs=5764 cs=0 l=1 rev1=1 crypto rx=0x7fce040a4dd0 tx=0x7fcdfc01c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ace17d155c32e73d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 1 -- 172.21.15.138:0/1282362454 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0ac070 con 0x7fce0406e0e0 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: dump: 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:02.724 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: _finish_auth 0 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: _check_auth_tickets 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 1 -- 172.21.15.138:0/1282362454 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdfc0a6660 con 0x7fce0406e0e0 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce09151640 5 monclient: authenticate success, global_id 22631 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 1 -- 172.21.15.138:0/1282362454 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0a6a30 con 0x7fce0406e0e0 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce09151640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce09151640 4 set_mon_vals no callback set 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.714+0000 7fce01d9b640 10 monclient: dump: 2024-09-11T05:14:02.725 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient: shutdown 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- 172.21.15.138:0/1282362454 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 msgr2=0x7fce04136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 --2- 172.21.15.138:0/1282362454 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce04136720 secure :-1 s=READY pgs=5764 cs=0 l=1 rev1=1 crypto rx=0x7fce040a4dd0 tx=0x7fcdfc01c7f0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- 172.21.15.138:0/1282362454 shutdown_connections 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 --2- 172.21.15.138:0/1282362454 >> v2:172.21.15.138:3300/0 conn(0x7fce040a1920 0x7fce040a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 --2- 172.21.15.138:0/1282362454 >> v2:172.21.15.190:3300/0 conn(0x7fce04136170 0x7fce04136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 --2- 172.21.15.138:0/1282362454 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce04136720 unknown :-1 s=CLOSED pgs=5764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- 172.21.15.138:0/1282362454 >> 172.21.15.138:0/1282362454 conn(0x7fce04063640 msgr2=0x7fce04063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- 172.21.15.138:0/1282362454 shutdown_connections 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- 172.21.15.138:0/1282362454 wait complete. 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient: build_initial_monmap 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient: monmap: 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:02.727 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 librados: starting msgr at 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 librados: starting objecter 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 Processor -- start 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- start start 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 librados: setting wanted keys 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 librados: calling monclient init 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient: init 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:14:02.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce041ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient(hunting): picked mon.noname-c con 0x7fce0406e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce040a1920 0x7fce0413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient(hunting): picked mon.noname-a con 0x7fce040a1920 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce04136170 0x7fce04136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient(hunting): picked mon.noname-b con 0x7fce04136170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fce04140090 con 0x7fce040a1920 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fce0413ff10 con 0x7fce04136170 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fce04140210 con 0x7fce0406e0e0 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce09151640 10 monclient(hunting): authenticate will time out at 1326.158023s 2024-09-11T05:14:02.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce041ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce040a1920 0x7fce0413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce041ce920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:53974/0 (socket says 172.21.15.138:53974) 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 1 -- 172.21.15.138:0/1389441183 learned_addr learned my addr 172.21.15.138:0/1389441183 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 10 monclient(hunting): get_auth_request con 0x7fce0406e0e0 auth_method 0 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient(hunting): get_auth_request con 0x7fce040a1920 auth_method 0 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:02.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient(hunting): handle_auth_done global_id 64299 payload 824 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0359e640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce04136170 0x7fce04136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 1 -- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 msgr2=0x7fce041ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce041ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 1 -- 172.21.15.138:0/1389441183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce04136170 msgr2=0x7fce04136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce04136170 0x7fce04136b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient: _finish_hunting 0 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 1 monclient: found mon.noname-a 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:02.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce0259c640 1 -- 172.21.15.138:0/1389441183 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce04133480 con 0x7fce040a1920 2024-09-11T05:14:02.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.718+0000 7fce02d9d640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce041ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:02.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce0259c640 1 -- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce040a1920 msgr2=0x7fce0413bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:02.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce0259c640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce040a1920 0x7fce0413bd10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c7c0 tx=0x7fcdec00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:02.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 client.?.objecter ms_handle_connect 0x7fce040a1920 2024-09-11T05:14:02.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:02.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 1 -- 172.21.15.138:0/1389441183 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec015070 con 0x7fce040a1920 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: dump: 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:02.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: _finish_auth 0 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 5 monclient: authenticate success, global_id 64299 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 1 -- 172.21.15.138:0/1389441183 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec004220 con 0x7fce040a1920 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 10 monclient: _renew_subs 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 1 -- 172.21.15.138:0/1389441183 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec007830 con 0x7fce040a1920 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce0413a500 con 0x7fce040a1920 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 monclient: dump: 2024-09-11T05:14:02.736 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce0159a640 4 set_mon_vals no callback set 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 10 client.64299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 10 monclient: _renew_subs 2024-09-11T05:14:02.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:02.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce0413a7e0 con 0x7fce040a1920 2024-09-11T05:14:02.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 1 librados: init done 2024-09-11T05:14:02.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:02.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:02.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:02.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce0409e110 con 0x7fce040a1920 2024-09-11T05:14:02.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 1 -- 172.21.15.138:0/1389441183 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fcdec0079d0 con 0x7fce040a1920 2024-09-11T05:14:02.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 1 --2- 172.21.15.138:0/1389441183 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcde803cef0 0x7fcde803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:02.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 1 -- 172.21.15.138:0/1389441183 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(310..310 src has 1..310) ==== 4989+0+0 (secure 0 0 0) 0x7fcdec04f660 con 0x7fce040a1920 2024-09-11T05:14:02.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 10 client.64299.objecter ms_dispatch 0x7fce04000ba0 osd_map(310..310 src has 1..310) 2024-09-11T05:14:02.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 3 client.64299.objecter handle_osd_map got epochs [310,310] > 0 2024-09-11T05:14:02.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 3 client.64299.objecter handle_osd_map decoding full epoch 310 2024-09-11T05:14:02.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fcdfb7fe640 20 client.64299.objecter dump_active .. 0 homeless 2024-09-11T05:14:02.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce02d9d640 1 --2- 172.21.15.138:0/1389441183 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcde803cef0 0x7fcde803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:02.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.722+0000 7fce02d9d640 10 monclient: get_auth_request con 0x7fcde803cef0 auth_method 0 2024-09-11T05:14:02.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.726+0000 7fcdfb7fe640 10 client.64299.objecter ms_handle_connect 0x7fcde803cef0 2024-09-11T05:14:02.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.726+0000 7fce02d9d640 1 --2- 172.21.15.138:0/1389441183 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcde803cef0 0x7fcde803f3b0 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc01c830 tx=0x7fcdfc0b4040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:02.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.726+0000 7fcdfb7fe640 1 -- 172.21.15.138:0/1389441183 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fcdec007cf0 con 0x7fce040a1920 2024-09-11T05:14:02.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.726+0000 7fcdfb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:02.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.726+0000 7fcdfb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:02.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.962+0000 7fce09151640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:14:02.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.962+0000 7fce09151640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:02.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.962+0000 7fce09151640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:02.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.962+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fce041348f0 con 0x7fce040a1920 2024-09-11T05:14:02.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.962+0000 7fcdfb7fe640 1 -- 172.21.15.138:0/1389441183 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fcdec003b40 con 0x7fce040a1920 2024-09-11T05:14:02.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.962+0000 7fcdfb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:02.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.962+0000 7fcdfb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:02.966 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":101,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:14:02.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 10 librados: watch_flush enter 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 10 librados: watch_flush exit 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 20 client.64299.objecter shutdown clearing up homeless session... 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 10 client.64299.objecter successfully canceled tick 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcde803cef0 msgr2=0x7fcde803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 --2- 172.21.15.138:0/1389441183 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcde803cef0 0x7fcde803f3b0 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc01c830 tx=0x7fcdfc0b4040 comp rx=0 tx=0).stop 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 10 monclient: shutdown 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce040a1920 msgr2=0x7fce0413bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce040a1920 0x7fce0413bd10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c7c0 tx=0x7fcdec00cc80 comp rx=0 tx=0).stop 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 shutdown_connections 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 --2- 172.21.15.138:0/1389441183 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcde803cef0 0x7fcde803f3b0 unknown :-1 s=CLOSED pgs=3397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fce04136170 0x7fce04136b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fce040a1920 0x7fce0413bd10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 --2- 172.21.15.138:0/1389441183 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fce0406e0e0 0x7fce041ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:02.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 >> 172.21.15.138:0/1389441183 conn(0x7fce04063640 msgr2=0x7fce0413c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:02.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 shutdown_connections 2024-09-11T05:14:02.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 -- 172.21.15.138:0/1389441183 wait complete. 2024-09-11T05:14:02.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:02.966+0000 7fce09151640 1 librados: shutdown 2024-09-11T05:14:05.987 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:14:06.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient: get_monmap_and_config 2024-09-11T05:14:06.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient: build_initial_monmap 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient: monmap: 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:06.119296+0000 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:06.119296+0000 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:06.118 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 1 Processor -- start 2024-09-11T05:14:06.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 1 -- start start 2024-09-11T05:14:06.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient: init 2024-09-11T05:14:06.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:06.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-11T05:14:06.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f4fa406e0e0 0x7f4fa409e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient(hunting): picked mon.noname-a con 0x7f4fa406e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fa409e960 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient(hunting): picked mon.noname-b con 0x7f4fa409ed70 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fa406e4c0 con 0x7f4fa409e960 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f4fa409f820 con 0x7f4fa406e0e0 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4fa409f540 con 0x7f4fa409ed70 2024-09-11T05:14:06.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4faa434640 10 monclient(hunting): authenticate will time out at 1329.555485s 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f4fa406e0e0 0x7f4fa409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f4fa406e0e0 0x7f4fa409e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43922/0 (socket says 172.21.15.138:43922) 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 1 -- 172.21.15.138:0/3183307776 learned_addr learned my addr 172.21.15.138:0/3183307776 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 10 monclient(hunting): get_auth_request con 0x7f4fa406e0e0 auth_method 0 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 10 monclient(hunting): get_auth_request con 0x7f4fa409ed70 auth_method 0 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 10 monclient(hunting): handle_auth_done global_id 22637 payload 274 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.114+0000 7f4fa89d1640 1 -- 172.21.15.138:0/3183307776 >> v2:172.21.15.138:3300/0 conn(0x7f4fa406e0e0 msgr2=0x7f4fa409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.118+0000 7f4fa89d1640 1 --2- 172.21.15.138:0/3183307776 >> v2:172.21.15.138:3300/0 conn(0x7f4fa406e0e0 0x7f4fa409e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.118+0000 7f4fa89d1640 1 -- 172.21.15.138:0/3183307776 >> v1:172.21.15.190:6789/0 conn(0x7f4fa409e960 legacy=0x7f4fa41356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:06.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.118+0000 7f4fa89d1640 10 monclient: _finish_hunting 0 2024-09-11T05:14:06.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.118+0000 7f4fa89d1640 1 monclient: found mon.noname-b 2024-09-11T05:14:06.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.118+0000 7f4fa89d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:06.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.118+0000 7f4fa89d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.118+0000 7f4fa89d1640 1 -- 172.21.15.138:0/3183307776 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa409fb00 con 0x7f4fa409ed70 2024-09-11T05:14:06.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.118+0000 7f4fa3fff640 1 --2- 172.21.15.138:0/3183307776 >> v2:172.21.15.138:3300/0 conn(0x7f4fa406e0e0 0x7f4fa409e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 1 -- 172.21.15.138:0/3183307776 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 msgr2=0x7f4fa4135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 1 --2- 172.21.15.138:0/3183307776 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135e00 secure :-1 s=READY pgs=5770 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0097c0 tx=0x7f4f9c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccabac4402c9fa16 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 1 -- 172.21.15.138:0/3183307776 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c0b0070 con 0x7f4fa409ed70 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: dump: 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.131 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.132 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.132 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.132 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.132 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.132 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.132 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: _finish_auth 0 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 1 -- 172.21.15.138:0/3183307776 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9c0ab600 con 0x7f4fa409ed70 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 5 monclient: authenticate success, global_id 22637 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 4 set_mon_vals no callback set 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 1 -- 172.21.15.138:0/3183307776 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c0ab9d0 con 0x7f4fa409ed70 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:06.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa2ffd640 10 monclient: dump: 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient: shutdown 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:06.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- 172.21.15.138:0/3183307776 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 msgr2=0x7f4fa4135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 --2- 172.21.15.138:0/3183307776 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135e00 secure :-1 s=READY pgs=5770 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0097c0 tx=0x7f4f9c01c360 comp rx=0 tx=0).stop 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- 172.21.15.138:0/3183307776 shutdown_connections 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 --2- 172.21.15.138:0/3183307776 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135e00 unknown :-1 s=CLOSED pgs=5770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 --2- 172.21.15.138:0/3183307776 >> v2:172.21.15.138:3300/0 conn(0x7f4fa406e0e0 0x7f4fa409e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- 172.21.15.138:0/3183307776 >> 172.21.15.138:0/3183307776 conn(0x7f4fa4063640 msgr2=0x7f4fa4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- 172.21.15.138:0/3183307776 shutdown_connections 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- 172.21.15.138:0/3183307776 wait complete. 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient: build_initial_monmap 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient: monmap: 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:06.135 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 librados: starting msgr at 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 librados: starting objecter 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 Processor -- start 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- start start 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 librados: setting wanted keys 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 librados: calling monclient init 2024-09-11T05:14:06.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient: init 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4fa406e0e0 0x7f4fa411d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient(hunting): picked mon.noname-a con 0x7f4fa406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fa409e960 0x7f4fa413b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient(hunting): picked mon.noname-b con 0x7f4fa409e960 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient(hunting): picked mon.noname-c con 0x7f4fa409ed70 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa41364c0 con 0x7f4fa406e0e0 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa4136340 con 0x7f4fa409e960 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4fa40a21b0 con 0x7f4fa409ed70 2024-09-11T05:14:06.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4faa434640 10 monclient(hunting): authenticate will time out at 1329.562759s 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4fa406e0e0 0x7f4fa411d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4fa406e0e0 0x7f4fa411d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43932/0 (socket says 172.21.15.138:43932) 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 -- 172.21.15.138:0/1700284764 learned_addr learned my addr 172.21.15.138:0/1700284764 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient(hunting): get_auth_request con 0x7f4fa406e0e0 auth_method 0 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 10 monclient(hunting): get_auth_request con 0x7f4fa409ed70 auth_method 0 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa89d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient(hunting): handle_auth_done global_id 64305 payload 824 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa37fe640 1 --2- 172.21.15.138:0/1700284764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fa409e960 0x7f4fa413b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 -- 172.21.15.138:0/1700284764 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 msgr2=0x7f4fa4135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 --2- 172.21.15.138:0/1700284764 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 -- 172.21.15.138:0/1700284764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fa409e960 msgr2=0x7f4fa413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 --2- 172.21.15.138:0/1700284764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fa409e960 0x7f4fa413b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient: _finish_hunting 0 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 monclient: found mon.noname-a 2024-09-11T05:14:06.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.122+0000 7f4fa3fff640 1 -- 172.21.15.138:0/1700284764 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa409e110 con 0x7f4fa406e0e0 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa3fff640 1 -- 172.21.15.138:0/1700284764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4fa406e0e0 msgr2=0x7f4fa411d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa3fff640 1 --2- 172.21.15.138:0/1700284764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4fa406e0e0 0x7f4fa411d5b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400efc0 tx=0x7f4f9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 client.?.objecter ms_handle_connect 0x7f4fa406e0e0 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 1 -- 172.21.15.138:0/1700284764 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94014680 con 0x7f4fa406e0e0 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: dump: 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.140 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: _finish_auth 0 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 5 monclient: authenticate success, global_id 64305 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 10 monclient: _renew_subs 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 1 -- 172.21.15.138:0/1700284764 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f94019070 con 0x7f4fa406e0e0 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa413ce00 con 0x7f4fa406e0e0 2024-09-11T05:14:06.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 1 -- 172.21.15.138:0/1700284764 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94014a20 con 0x7f4fa406e0e0 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa27fc640 4 set_mon_vals no callback set 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 monclient: dump: 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 10 client.64305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 10 monclient: _renew_subs 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa411e510 con 0x7f4fa406e0e0 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 1 librados: init done 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fa409e420 con 0x7f4fa406e0e0 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 1 -- 172.21.15.138:0/1700284764 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f4f94014bc0 con 0x7f4fa406e0e0 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 1 --2- 172.21.15.138:0/1700284764 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f8403ced0 0x7f4f8403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 1 -- 172.21.15.138:0/1700284764 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(313..313 src has 1..313) ==== 4989+0+0 (secure 0 0 0) 0x7f4f940500e0 con 0x7f4fa406e0e0 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 10 client.64305.objecter ms_dispatch 0x7f4fa4000ba0 osd_map(313..313 src has 1..313) 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 3 client.64305.objecter handle_osd_map got epochs [313,313] > 0 2024-09-11T05:14:06.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 3 client.64305.objecter handle_osd_map decoding full epoch 313 2024-09-11T05:14:06.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa0ff9640 20 client.64305.objecter dump_active .. 0 homeless 2024-09-11T05:14:06.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa37fe640 1 --2- 172.21.15.138:0/1700284764 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f8403ced0 0x7f4f8403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.126+0000 7f4fa37fe640 10 monclient: get_auth_request con 0x7f4f8403ced0 auth_method 0 2024-09-11T05:14:06.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.130+0000 7f4fa37fe640 1 --2- 172.21.15.138:0/1700284764 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f8403ced0 0x7f4f8403f390 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4072ff0 tx=0x7f4f8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.130+0000 7f4fa0ff9640 10 client.64305.objecter ms_handle_connect 0x7f4f8403ced0 2024-09-11T05:14:06.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.130+0000 7f4fa0ff9640 1 -- 172.21.15.138:0/1700284764 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f4f94010040 con 0x7f4fa406e0e0 2024-09-11T05:14:06.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.130+0000 7f4fa0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:06.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.130+0000 7f4fa0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:06.367 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:14:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4fa411e180 con 0x7f4fa406e0e0 2024-09-11T05:14:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4fa0ff9640 1 -- 172.21.15.138:0/1700284764 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f4f9401bd20 con 0x7f4fa406e0e0 2024-09-11T05:14:06.368 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4fa0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:06.369 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4fa0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:06.369 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:14:06.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 10 librados: watch_flush enter 2024-09-11T05:14:06.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 10 librados: watch_flush exit 2024-09-11T05:14:06.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 20 client.64305.objecter shutdown clearing up homeless session... 2024-09-11T05:14:06.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 10 client.64305.objecter successfully canceled tick 2024-09-11T05:14:06.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f8403ced0 msgr2=0x7f4f8403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 --2- 172.21.15.138:0/1700284764 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f8403ced0 0x7f4f8403f390 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4072ff0 tx=0x7f4f8c009040 comp rx=0 tx=0).stop 2024-09-11T05:14:06.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 10 monclient: shutdown 2024-09-11T05:14:06.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4fa406e0e0 msgr2=0x7f4fa411d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 --2- 172.21.15.138:0/1700284764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4fa406e0e0 0x7f4fa411d5b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400efc0 tx=0x7f4f9400c490 comp rx=0 tx=0).stop 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 shutdown_connections 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 --2- 172.21.15.138:0/1700284764 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4f8403ced0 0x7f4f8403f390 unknown :-1 s=CLOSED pgs=3398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 --2- 172.21.15.138:0/1700284764 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4fa409ed70 0x7f4fa4135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 --2- 172.21.15.138:0/1700284764 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4fa409e960 0x7f4fa413b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 --2- 172.21.15.138:0/1700284764 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4fa406e0e0 0x7f4fa411d5b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 >> 172.21.15.138:0/1700284764 conn(0x7f4fa4063640 msgr2=0x7f4fa40a56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 shutdown_connections 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 -- 172.21.15.138:0/1700284764 wait complete. 2024-09-11T05:14:06.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.366+0000 7f4faa434640 1 librados: shutdown 2024-09-11T05:14:06.387 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:14:06.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:14:06.388 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:14:06.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.502+0000 7efdf1603640 10 monclient: get_monmap_and_config 2024-09-11T05:14:06.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.502+0000 7efdf1603640 10 monclient: build_initial_monmap 2024-09-11T05:14:06.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.502+0000 7efdf1603640 10 monclient: monmap: 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:06.508274+0000 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:06.508274+0000 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:06.507 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 1 Processor -- start 2024-09-11T05:14:06.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 1 -- start start 2024-09-11T05:14:06.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient: init 2024-09-11T05:14:06.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:06.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-11T05:14:06.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec070010 0x7efdec135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient(hunting): picked mon.noname-b con 0x7efdec070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:06.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efdec09e6e0 0x7efdec09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient(hunting): picked mon.noname-c con 0x7efdec09e6e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdec1382c0 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdec0a8990 con 0x7efdec1382c0 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efdec0703f0 con 0x7efdec09e6e0 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7efdec09cbf0 con 0x7efdec070010 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdf1603640 10 monclient(hunting): authenticate will time out at 1329.944778s 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec070010 0x7efdec135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec070010 0x7efdec135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:54004/0 (socket says 172.21.15.138:54004) 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 1 -- 172.21.15.138:0/113881017 learned_addr learned my addr 172.21.15.138:0/113881017 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient(hunting): get_auth_request con 0x7efdec070010 auth_method 0 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient(hunting): handle_auth_done global_id 22643 payload 274 2024-09-11T05:14:06.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 1 -- 172.21.15.138:0/113881017 >> v2:172.21.15.190:3300/0 conn(0x7efdec09e6e0 msgr2=0x7efdec09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:06.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 1 --2- 172.21.15.138:0/113881017 >> v2:172.21.15.190:3300/0 conn(0x7efdec09e6e0 0x7efdec09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 1 -- 172.21.15.138:0/113881017 >> v1:172.21.15.190:6789/0 conn(0x7efdec1382c0 legacy=0x7efdec138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:06.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient: _finish_hunting 0 2024-09-11T05:14:06.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 1 monclient: found mon.noname-b 2024-09-11T05:14:06.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:06.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.506+0000 7efdeaffd640 1 -- 172.21.15.138:0/113881017 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdec09ced0 con 0x7efdec070010 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdeaffd640 1 -- 172.21.15.138:0/113881017 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec070010 msgr2=0x7efdec135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdeaffd640 1 --2- 172.21.15.138:0/113881017 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec070010 0x7efdec135450 secure :-1 s=READY pgs=5771 cs=0 l=1 rev1=1 crypto rx=0x7efddc009870 tx=0x7efddc01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a81a1e1ef4f94b95 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 1 -- 172.21.15.138:0/113881017 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc0b2070 con 0x7efdec070010 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: dump: 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.517 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: _finish_auth 0 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 1 -- 172.21.15.138:0/113881017 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efddc0ad640 con 0x7efdec070010 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 1 -- 172.21.15.138:0/113881017 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc0ada10 con 0x7efdec070010 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 5 monclient: authenticate success, global_id 22643 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 4 set_mon_vals no callback set 2024-09-11T05:14:06.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efde9ffb640 10 monclient: dump: 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient: shutdown 2024-09-11T05:14:06.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- 172.21.15.138:0/113881017 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec070010 msgr2=0x7efdec135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 --2- 172.21.15.138:0/113881017 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec070010 0x7efdec135450 secure :-1 s=READY pgs=5771 cs=0 l=1 rev1=1 crypto rx=0x7efddc009870 tx=0x7efddc01c3f0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- 172.21.15.138:0/113881017 shutdown_connections 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 --2- 172.21.15.138:0/113881017 >> v2:172.21.15.190:3300/0 conn(0x7efdec09e6e0 0x7efdec09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 --2- 172.21.15.138:0/113881017 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec070010 0x7efdec135450 unknown :-1 s=CLOSED pgs=5771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- 172.21.15.138:0/113881017 >> 172.21.15.138:0/113881017 conn(0x7efdec0ab830 msgr2=0x7efdec0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- 172.21.15.138:0/113881017 shutdown_connections 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- 172.21.15.138:0/113881017 wait complete. 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient: build_initial_monmap 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient: monmap: 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:06.520 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 librados: starting msgr at 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 librados: starting objecter 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 Processor -- start 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- start start 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 librados: setting wanted keys 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 librados: calling monclient init 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient: init 2024-09-11T05:14:06.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec070010 0x7efdec13f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient(hunting): picked mon.noname-b con 0x7efdec070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec09e6e0 0x7efdec13fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient(hunting): picked mon.noname-a con 0x7efdec09e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec1382c0 0x7efdec0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient(hunting): picked mon.noname-c con 0x7efdec1382c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efdec1432c0 con 0x7efdec09e6e0 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efdec143140 con 0x7efdec070010 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7efdec143440 con 0x7efdec1382c0 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:06.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdf1603640 10 monclient(hunting): authenticate will time out at 1329.951782s 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec09e6e0 0x7efdec13fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdeb7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec1382c0 0x7efdec0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec09e6e0 0x7efdec13fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43948/0 (socket says 172.21.15.138:43948) 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdeb7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec1382c0 0x7efdec0a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:54010/0 (socket says 172.21.15.138:54010) 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 1 -- 172.21.15.138:0/2069548357 learned_addr learned my addr 172.21.15.138:0/2069548357 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 10 monclient(hunting): get_auth_request con 0x7efdec09e6e0 auth_method 0 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdeb7fe640 10 monclient(hunting): get_auth_request con 0x7efdec1382c0 auth_method 0 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.510+0000 7efdea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec070010 0x7efdec13f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 10 monclient(hunting): handle_auth_done global_id 22646 payload 824 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec070010 msgr2=0x7efdec13f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec070010 0x7efdec13f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec09e6e0 msgr2=0x7efdec13fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec09e6e0 0x7efdec13fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:14:06.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 1 monclient: found mon.noname-c 2024-09-11T05:14:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 1 -- 172.21.15.138:0/2069548357 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdec13b1d0 con 0x7efdec1382c0 2024-09-11T05:14:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient: get_auth_request con 0x7efdec070010 auth_method 0 2024-09-11T05:14:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec070010 0x7efdec13f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:14:06.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdea7fc640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec09e6e0 0x7efdec13fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:06.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec1382c0 msgr2=0x7efdec0a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeb7fe640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec1382c0 0x7efdec0a38b0 secure :-1 s=READY pgs=5772 cs=0 l=1 rev1=1 crypto rx=0x7efde400c780 tx=0x7efde400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 client.?.objecter ms_handle_connect 0x7efdec1382c0 2024-09-11T05:14:06.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:06.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde4017070 con 0x7efdec1382c0 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: dump: 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.530 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: _finish_auth 0 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: _check_auth_tickets 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde400ce30 con 0x7efdec1382c0 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 5 monclient: authenticate success, global_id 22646 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 10 monclient: _renew_subs 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdec13a930 con 0x7efdec1382c0 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde4005290 con 0x7efdec1382c0 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 10 monclient: dump: 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.531 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efde97fa640 4 set_mon_vals no callback set 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 10 client.22646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 10 monclient: _renew_subs 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efdec140300 con 0x7efdec1382c0 2024-09-11T05:14:06.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 1 librados: init done 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec1d0ae0 0x7efdec0a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efdec139a90 con 0x7efdec1d0ae0 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec1d0ae0 0x7efdec0a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient: get_auth_request con 0x7efdec1d0ae0 auth_method 0 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7efde40054b0 con 0x7efdec1382c0 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 10 monclient(hunting): handle_auth_done global_id 64311 payload 824 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdeaffd640 1 -- 172.21.15.138:0/2069548357 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efdec135450 con 0x7efdec1d0ae0 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.514+0000 7efdd37fe640 1 --2- 172.21.15.138:0/2069548357 >> v2:172.21.15.190:6800/1562782189 conn(0x7efdd803cf20 0x7efdd803f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(314..314 src has 1..314) ==== 5361+0+0 (secure 0 0 0) 0x7efde404f2c0 con 0x7efdec1382c0 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 client.22646.objecter ms_dispatch 0x7efdec000ba0 osd_map(314..314 src has 1..314) 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 3 client.22646.objecter handle_osd_map got epochs [314,314] > 0 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 3 client.22646.objecter handle_osd_map decoding full epoch 314 2024-09-11T05:14:06.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 20 client.22646.objecter dump_active .. 0 homeless 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdea7fc640 1 --2- 172.21.15.138:0/2069548357 >> v2:172.21.15.190:6800/1562782189 conn(0x7efdd803cf20 0x7efdd803f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdea7fc640 10 monclient: get_auth_request con 0x7efdd803cf20 auth_method 0 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdeaffd640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec1d0ae0 msgr2=0x7efdec0a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 client.22646.objecter ms_handle_connect 0x7efdec1d0ae0 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 client.22646.objecter resend_mon_ops 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdeaffd640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec1d0ae0 0x7efdec0a3df0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdec0a7a00 tx=0x7efddc019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc0bb020 con 0x7efdec1d0ae0 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 monclient: dump: 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.535 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efddc0b3040 con 0x7efdec1d0ae0 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec1d0ae0 msgr2=0x7efdec0a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec1d0ae0 0x7efdec0a3df0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdec0a7a00 tx=0x7efddc019190 comp rx=0 tx=0).stop 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec1d0ae0 msgr2=0x7efdec0a3df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:06.536 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec1d0ae0 0x7efdec0a3df0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdec0a7a00 tx=0x7efddc019190 comp rx=0 tx=0).stop 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdd37fe640 10 client.22646.objecter ms_handle_connect 0x7efdd803cf20 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.518+0000 7efdea7fc640 1 --2- 172.21.15.138:0/2069548357 >> v2:172.21.15.190:6800/1562782189 conn(0x7efdd803cf20 0x7efdd803f3e0 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7efdec13a520 tx=0x7efde00091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec0a4330 0x7efdec09f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efdec0a4740 con 0x7efdec0a4330 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec0a4330 0x7efdec09f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 10 monclient: get_auth_request con 0x7efdec0a4330 auth_method 0 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.537 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 10 monclient(hunting): handle_auth_done global_id 64314 payload 824 2024-09-11T05:14:06.538 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 1 -- 172.21.15.138:0/2069548357 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efdec13ae50 con 0x7efdec0a4330 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 10 client.22646.objecter ms_handle_connect 0x7efdec0a4330 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 10 client.22646.objecter resend_mon_ops 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec0a4330 msgr2=0x7efdec09f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeaffd640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec0a4330 0x7efdec09f0b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efddc01c220 tx=0x7efddc0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc0bb070 con 0x7efdec0a4330 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 10 monclient: dump: 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.539 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7efddc0b7a10 con 0x7efdec0a4330 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec0a4330 msgr2=0x7efdec09f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec0a4330 0x7efdec09f0b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efddc01c220 tx=0x7efddc0a7000 comp rx=0 tx=0).stop 2024-09-11T05:14:06.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec0a4330 msgr2=0x7efdec09f0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdd37fe640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efdec0a4330 0x7efdec09f0b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 26, 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 150156, 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:14:06.541 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 3, 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:06.542 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:06.543 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:06.544 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:06.545 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:06.546 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:06.548 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:14:06.549 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.550 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.551 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:14:06.553 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:14:06.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 10 librados: watch_flush enter 2024-09-11T05:14:06.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 10 librados: watch_flush exit 2024-09-11T05:14:06.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 20 client.22646.objecter shutdown clearing up homeless session... 2024-09-11T05:14:06.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 10 client.22646.objecter successfully canceled tick 2024-09-11T05:14:06.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 >> v2:172.21.15.190:6800/1562782189 conn(0x7efdd803cf20 msgr2=0x7efdd803f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 --2- 172.21.15.138:0/2069548357 >> v2:172.21.15.190:6800/1562782189 conn(0x7efdd803cf20 0x7efdd803f3e0 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7efdec13a520 tx=0x7efde00091c0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 10 monclient: shutdown 2024-09-11T05:14:06.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec1382c0 msgr2=0x7efdec0a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdeb7fe640 1 -- 172.21.15.138:0/2069548357 reap_dead start 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec1382c0 0x7efdec0a38b0 secure :-1 s=READY pgs=5772 cs=0 l=1 rev1=1 crypto rx=0x7efde400c780 tx=0x7efde400cc40 comp rx=0 tx=0).stop 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 shutdown_connections 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 --2- 172.21.15.138:0/2069548357 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efdec1382c0 0x7efdec0a38b0 unknown :-1 s=CLOSED pgs=5772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 >> 172.21.15.138:0/2069548357 conn(0x7efdec0ab830 msgr2=0x7efdec06efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 shutdown_connections 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.526+0000 7efdf1603640 1 -- 172.21.15.138:0/2069548357 wait complete. 2024-09-11T05:14:06.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.530+0000 7efdf1603640 1 librados: shutdown 2024-09-11T05:14:06.556 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:14:06.557 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:14:06.557 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient: get_monmap_and_config 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient: build_initial_monmap 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient: monmap: 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:06.669646+0000 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:06.669646+0000 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:06.668 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:06.669 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:06.669 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:06.669 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:06.669 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 1 Processor -- start 2024-09-11T05:14:06.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 1 -- start start 2024-09-11T05:14:06.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient: init 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d70070010 addr v1:172.21.15.138:6789/0 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7009e6e0 0x7f8d7009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7009e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d7013d640 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d70135b60 con 0x7f8d70070010 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d700a8990 con 0x7f8d7013d640 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8d700703f0 con 0x7f8d7009e6e0 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d76fbe640 10 monclient(hunting): authenticate will time out at 1330.105647s 2024-09-11T05:14:06.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d74d5a640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f8d70070010 0x7f8d70135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:55398/0 (socket says 172.21.15.138:55398) 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d74d5a640 1 -- 172.21.15.138:0/2331889181 learned_addr learned my addr 172.21.15.138:0/2331889181 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 1 --2- 172.21.15.138:0/2331889181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7009e6e0 0x7f8d7009eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient(hunting): get_auth_request con 0x7f8d7009e6e0 auth_method 0 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_done global_id 22649 payload 274 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 1 -- 172.21.15.138:0/2331889181 >> v1:172.21.15.190:6789/0 conn(0x7f8d7013d640 legacy=0x7f8d7013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 1 -- 172.21.15.138:0/2331889181 >> v1:172.21.15.138:6789/0 conn(0x7f8d70070010 legacy=0x7f8d70135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:06.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:14:06.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 1 monclient: found mon.noname-b 2024-09-11T05:14:06.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:06.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.666+0000 7f8d6ffff640 1 -- 172.21.15.138:0/2331889181 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d7009cc40 con 0x7f8d7009e6e0 2024-09-11T05:14:06.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6ffff640 1 -- 172.21.15.138:0/2331889181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7009e6e0 msgr2=0x7f8d7009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6ffff640 1 --2- 172.21.15.138:0/2331889181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7009e6e0 0x7f8d7009eae0 secure :-1 s=READY pgs=5773 cs=0 l=1 rev1=1 crypto rx=0x7f8d640097f0 tx=0x7f8d6401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35de79486a63388 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 1 -- 172.21.15.138:0/2331889181 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d640b0070 con 0x7f8d7009e6e0 2024-09-11T05:14:06.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:14:06.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: dump: 2024-09-11T05:14:06.674 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.676 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.676 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.676 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.676 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.676 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.676 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.676 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 1 -- 172.21.15.138:0/2331889181 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d640ab600 con 0x7f8d7009e6e0 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 1 -- 172.21.15.138:0/2331889181 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d640a4e60 con 0x7f8d7009e6e0 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 5 monclient: authenticate success, global_id 22649 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 4 set_mon_vals no callback set 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:06.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d6f7fe640 10 monclient: dump: 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.678 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 10 monclient: shutdown 2024-09-11T05:14:06.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:06.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 1 -- 172.21.15.138:0/2331889181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7009e6e0 msgr2=0x7f8d7009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 1 --2- 172.21.15.138:0/2331889181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7009e6e0 0x7f8d7009eae0 secure :-1 s=READY pgs=5773 cs=0 l=1 rev1=1 crypto rx=0x7f8d640097f0 tx=0x7f8d6401c3a0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 1 -- 172.21.15.138:0/2331889181 shutdown_connections 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 1 --2- 172.21.15.138:0/2331889181 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7009e6e0 0x7f8d7009eae0 unknown :-1 s=CLOSED pgs=5773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 1 -- 172.21.15.138:0/2331889181 >> 172.21.15.138:0/2331889181 conn(0x7f8d700ab830 msgr2=0x7f8d700a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 1 -- 172.21.15.138:0/2331889181 shutdown_connections 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 1 -- 172.21.15.138:0/2331889181 wait complete. 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 10 monclient: build_initial_monmap 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.670+0000 7f8d76fbe640 10 monclient: monmap: 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:06.685 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 librados: starting msgr at 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 librados: starting objecter 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 Processor -- start 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 -- start start 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 librados: setting wanted keys 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 librados: calling monclient init 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient: init 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:14:06.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8d70070010 0x7f8d70132040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient(hunting): picked mon.noname-a con 0x7f8d70070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d7009e6e0 0x7f8d70136260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7009e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7013d640 0x7f8d701367a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient(hunting): picked mon.noname-c con 0x7f8d7013d640 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8d70143e10 con 0x7f8d70070010 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8d70143c90 con 0x7f8d7009e6e0 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8d70143f90 con 0x7f8d7013d640 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient(hunting): authenticate will time out at 1330.112846s 2024-09-11T05:14:06.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8d70070010 0x7f8d70132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7013d640 0x7f8d701367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8d70070010 0x7f8d70132040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43964/0 (socket says 172.21.15.138:43964) 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 -- 172.21.15.138:0/3685280421 learned_addr learned my addr 172.21.15.138:0/3685280421 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient(hunting): get_auth_request con 0x7f8d70070010 auth_method 0 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 10 monclient(hunting): get_auth_request con 0x7f8d7013d640 auth_method 0 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient(hunting): handle_auth_done global_id 64317 payload 824 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6ffff640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d7009e6e0 0x7f8d70136260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7013d640 msgr2=0x7f8d701367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7013d640 0x7f8d701367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d7009e6e0 msgr2=0x7f8d70136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d7009e6e0 0x7f8d70136260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient: _finish_hunting 0 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 monclient: found mon.noname-a 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:06.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 -- 172.21.15.138:0/3685280421 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d70137ba0 con 0x7f8d70070010 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d7555b640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d7013d640 0x7f8d701367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8d70070010 msgr2=0x7f8d70132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d74d5a640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8d70070010 0x7f8d70132040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c470 tx=0x7f8d6000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 client.?.objecter ms_handle_connect 0x7f8d70070010 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60017070 con 0x7f8d70070010 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: dump: 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.690 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: _finish_auth 0 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 5 monclient: authenticate success, global_id 64317 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient: _renew_subs 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d600070d0 con 0x7f8d70070010 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d700a2330 con 0x7f8d70070010 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d600074c0 con 0x7f8d70070010 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6d7fa640 10 monclient: dump: 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.692 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d6effd640 4 set_mon_vals no callback set 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 client.64317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient: _renew_subs 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d70132580 con 0x7f8d70070010 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.674+0000 7f8d76fbe640 1 librados: init done 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d76fbe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d76fbe640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d700a3eb0 0x7f8d70136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d76fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8d7009eae0 con 0x7f8d700a3eb0 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d700a3eb0 0x7f8d70136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 10 monclient: get_auth_request con 0x7f8d700a3eb0 auth_method 0 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_done global_id 22655 payload 824 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f8d6001b3b0 con 0x7f8d70070010 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 1 -- 172.21.15.138:0/3685280421 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d700a3380 con 0x7f8d700a3eb0 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 --2- 172.21.15.138:0/3685280421 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8d5c03cea0 0x7f8d5c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(314..314 src has 1..314) ==== 5361+0+0 (secure 0 0 0) 0x7f8d6004f2f0 con 0x7f8d70070010 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 10 client.64317.objecter ms_dispatch 0x7f8d70000ba0 osd_map(314..314 src has 1..314) 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 3 client.64317.objecter handle_osd_map got epochs [314,314] > 0 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 3 client.64317.objecter handle_osd_map decoding full epoch 314 2024-09-11T05:14:06.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 20 client.64317.objecter dump_active .. 0 homeless 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d7555b640 1 --2- 172.21.15.138:0/3685280421 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8d5c03cea0 0x7f8d5c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d7555b640 10 monclient: get_auth_request con 0x7f8d5c03cea0 auth_method 0 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d700a3eb0 msgr2=0x7f8d70136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 10 client.64317.objecter ms_handle_connect 0x7f8d700a3eb0 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 10 client.64317.objecter resend_mon_ops 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6ffff640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d700a3eb0 0x7f8d70136ce0 secure :-1 s=READY pgs=5774 cs=0 l=1 rev1=1 crypto rx=0x7f8d640ad000 tx=0x7f8d6401abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d640bb020 con 0x7f8d700a3eb0 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 10 monclient: dump: 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.695 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d640a4e60 con 0x7f8d700a3eb0 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d700a3eb0 msgr2=0x7f8d70136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d700a3eb0 0x7f8d70136ce0 secure :-1 s=READY pgs=5774 cs=0 l=1 rev1=1 crypto rx=0x7f8d640ad000 tx=0x7f8d6401abe0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d700a3eb0 msgr2=0x7f8d70136ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.678+0000 7f8d6d7fa640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d700a3eb0 0x7f8d70136ce0 secure :-1 s=CLOSED pgs=5774 cs=0 l=1 rev1=1 crypto rx=0x7f8d640ad000 tx=0x7f8d6401abe0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.682+0000 7f8d6d7fa640 10 client.64317.objecter ms_handle_connect 0x7f8d5c03cea0 2024-09-11T05:14:06.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.682+0000 7f8d7555b640 1 --2- 172.21.15.138:0/3685280421 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8d5c03cea0 0x7f8d5c03f360 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7f8d70135220 tx=0x7f8d68003d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d76fbe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:14:06.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d76fbe640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d70137220 0x7f8d700a9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d76fbe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8d70137600 con 0x7f8d70137220 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d70137220 0x7f8d700a9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 10 monclient: get_auth_request con 0x7f8d70137220 auth_method 0 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 10 monclient(hunting): handle_auth_done global_id 22658 payload 824 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.686+0000 7f8d6ffff640 1 -- 172.21.15.138:0/3685280421 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d701327b0 con 0x7f8d70137220 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6ffff640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d70137220 msgr2=0x7f8d700a9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 10 client.64317.objecter ms_handle_connect 0x7f8d70137220 2024-09-11T05:14:06.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6ffff640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d70137220 0x7f8d700a9570 secure :-1 s=READY pgs=5775 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009740 tx=0x7f8d64019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 10 client.64317.objecter resend_mon_ops 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d640bb070 con 0x7f8d70137220 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 10 monclient: dump: 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.700 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f8d640c2860 con 0x7f8d70137220 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d70137220 msgr2=0x7f8d700a9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d70137220 0x7f8d700a9570 secure :-1 s=READY pgs=5775 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009740 tx=0x7f8d64019140 comp rx=0 tx=0).stop 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d70137220 msgr2=0x7f8d700a9570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d6d7fa640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8d70137220 0x7f8d700a9570 secure :-1 s=CLOSED pgs=5775 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009740 tx=0x7f8d64019140 comp rx=0 tx=0).stop 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:14:06.701 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 26, 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 405463, 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 3, 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:14:06.702 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:14:06.703 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.704 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:14:06.705 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:06.706 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.709 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:06.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:14:06.711 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.712 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:06.713 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:14:06.714 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:14:06.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 10 librados: watch_flush enter 2024-09-11T05:14:06.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 10 librados: watch_flush exit 2024-09-11T05:14:06.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 20 client.64317.objecter shutdown clearing up homeless session... 2024-09-11T05:14:06.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 10 client.64317.objecter successfully canceled tick 2024-09-11T05:14:06.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8d5c03cea0 msgr2=0x7f8d5c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 --2- 172.21.15.138:0/3685280421 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8d5c03cea0 0x7f8d5c03f360 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7f8d70135220 tx=0x7f8d68003d50 comp rx=0 tx=0).stop 2024-09-11T05:14:06.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 10 monclient: shutdown 2024-09-11T05:14:06.716 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8d70070010 msgr2=0x7f8d70132040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 --2- 172.21.15.138:0/3685280421 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f8d70070010 0x7f8d70132040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c470 tx=0x7f8d6000c930 comp rx=0 tx=0).stop 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d7555b640 1 -- 172.21.15.138:0/3685280421 reap_dead start 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d7555b640 1 -- 172.21.15.138:0/3685280421 reap_dead start 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 shutdown_connections 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 >> 172.21.15.138:0/3685280421 conn(0x7f8d700ab830 msgr2=0x7f8d7006ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 shutdown_connections 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 -- 172.21.15.138:0/3685280421 wait complete. 2024-09-11T05:14:06.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.690+0000 7f8d76fbe640 1 librados: shutdown 2024-09-11T05:14:06.717 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:14:06.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:14:06.718 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:14:06.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient: get_monmap_and_config 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient: build_initial_monmap 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient: monmap: 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:06.830502+0000 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:06.830502+0000 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:06.829 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:06.830 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:06.830 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:06.830 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 1 Processor -- start 2024-09-11T05:14:06.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 1 -- start start 2024-09-11T05:14:06.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient: init 2024-09-11T05:14:06.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:06.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f650009da20 0x7f65000a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient(hunting): picked mon.noname-a con 0x7f650009da20 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65000a0a90 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6500139b70 0x7f650013bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient(hunting): picked mon.noname-b con 0x7f6500139b70 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6500064590 con 0x7f65000a0a90 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f65000a3880 con 0x7f650009da20 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f650009efa0 con 0x7f6500139b70 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f6505f50640 10 monclient(hunting): authenticate will time out at 1330.267357s 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f650009da20 0x7f65000a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6500139b70 0x7f650013bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f650009da20 0x7f65000a26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43968/0 (socket says 172.21.15.138:43968) 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6500139b70 0x7f650013bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:54036/0 (socket says 172.21.15.138:54036) 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 1 -- 172.21.15.138:0/2643574448 learned_addr learned my addr 172.21.15.138:0/2643574448 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient(hunting): get_auth_request con 0x7f650009da20 auth_method 0 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 10 monclient(hunting): get_auth_request con 0x7f6500139b70 auth_method 0 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient(hunting): handle_auth_done global_id 64320 payload 274 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 1 -- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6500139b70 msgr2=0x7f650013bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 1 --2- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6500139b70 0x7f650013bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 1 -- 172.21.15.138:0/2643574448 >> v1:172.21.15.190:6789/0 conn(0x7f65000a0a90 legacy=0x7f65000a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 1 monclient: found mon.noname-a 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64ff7fe640 1 -- 172.21.15.138:0/2643574448 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650006e240 con 0x7f650009da20 2024-09-11T05:14:06.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.826+0000 7f64fffff640 1 --2- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6500139b70 0x7f650013bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:06.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 -- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f650009da20 msgr2=0x7f65000a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 --2- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f650009da20 0x7f65000a26c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f64f00083f0 tx=0x7f64f001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198c68013bcf30cb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 1 -- 172.21.15.138:0/2643574448 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f000a2d0 con 0x7f650009da20 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: dump: 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.840 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: _finish_auth 0 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 5 monclient: authenticate success, global_id 64320 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 1 -- 172.21.15.138:0/2643574448 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64f00a5610 con 0x7f650009da20 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 4 set_mon_vals no callback set 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 1 -- 172.21.15.138:0/2643574448 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f00a59c0 con 0x7f650009da20 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fe7fc640 10 monclient: dump: 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient: shutdown 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:06.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f650009da20 msgr2=0x7f65000a26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 --2- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f650009da20 0x7f65000a26c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f64f00083f0 tx=0x7f64f001c940 comp rx=0 tx=0).stop 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- 172.21.15.138:0/2643574448 shutdown_connections 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 --2- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6500139b70 0x7f650013bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 --2- 172.21.15.138:0/2643574448 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f650009da20 0x7f65000a26c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- 172.21.15.138:0/2643574448 >> 172.21.15.138:0/2643574448 conn(0x7f6500063640 msgr2=0x7f6500063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- 172.21.15.138:0/2643574448 shutdown_connections 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- 172.21.15.138:0/2643574448 wait complete. 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient: build_initial_monmap 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient: monmap: 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:06.843 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 librados: starting msgr at 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 librados: starting objecter 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 Processor -- start 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- start start 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 librados: setting wanted keys 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 librados: calling monclient init 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient: init 2024-09-11T05:14:06.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f650009da20 0x7f65001cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient(hunting): picked mon.noname-c con 0x7f650009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f65000a0a90 0x7f6500134770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient(hunting): picked mon.noname-a con 0x7f65000a0a90 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500139b70 0x7f6500072220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient(hunting): picked mon.noname-b con 0x7f6500139b70 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f650013e940 con 0x7f65000a0a90 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f650013e7c0 con 0x7f6500139b70 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f650013eac0 con 0x7f650009da20 2024-09-11T05:14:06.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f6505f50640 10 monclient(hunting): authenticate will time out at 1330.274472s 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f650009da20 0x7f65001cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f65000a0a90 0x7f6500134770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f650009da20 0x7f65001cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:54040/0 (socket says 172.21.15.138:54040) 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 -- 172.21.15.138:0/35189681 learned_addr learned my addr 172.21.15.138:0/35189681 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient(hunting): get_auth_request con 0x7f650009da20 auth_method 0 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 10 monclient(hunting): get_auth_request con 0x7f65000a0a90 auth_method 0 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient(hunting): handle_auth_done global_id 22664 payload 824 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500139b70 msgr2=0x7f6500072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64fffff640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500139b70 0x7f6500072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500139b70 0x7f6500072220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f65000a0a90 msgr2=0x7f6500134770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f65000a0a90 0x7f6500134770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 monclient: found mon.noname-c 2024-09-11T05:14:06.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:06.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64ff7fe640 1 -- 172.21.15.138:0/35189681 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65001cccd0 con 0x7f650009da20 2024-09-11T05:14:06.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.834+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f65000a0a90 0x7f6500134770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:06.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64ff7fe640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f650009da20 msgr2=0x7f65001cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:06.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64ff7fe640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f650009da20 0x7f65001cbdc0 secure :-1 s=READY pgs=5776 cs=0 l=1 rev1=1 crypto rx=0x7f64f00af040 tx=0x7f64f000a870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 client.?.objecter ms_handle_connect 0x7f650009da20 2024-09-11T05:14:06.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:06.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f00bb070 con 0x7f650009da20 2024-09-11T05:14:06.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: dump: 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: _finish_auth 0 2024-09-11T05:14:06.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: _check_auth_tickets 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64f00a5bd0 con 0x7f650009da20 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 5 monclient: authenticate success, global_id 22664 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f0004040 con 0x7f650009da20 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 10 monclient: dump: 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:06.852 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 10 monclient: _renew_subs 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650011ee40 con 0x7f650009da20 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64fdffb640 4 set_mon_vals no callback set 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 10 client.22664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 10 monclient: _renew_subs 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f650011d920 con 0x7f650009da20 2024-09-11T05:14:06.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 1 librados: init done 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f650009e1a0 con 0x7f6500134cb0 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 10 monclient: get_auth_request con 0x7f6500134cb0 auth_method 0 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:06.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64ff7fe640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f650009da20 0x7f65001cbdc0 secure :-1 s=READY pgs=5776 cs=0 l=1 rev1=1 crypto rx=0x7f64f00af040 tx=0x7f64f000a870 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f64f00041e0 mgrmap(e 5) 2024-09-11T05:14:06.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f64f00041e0 con 0x7f650009da20 2024-09-11T05:14:06.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64e7fff640 1 --2- 172.21.15.138:0/35189681 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64ec03cea0 0x7f64ec03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:06.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64fffff640 1 --2- 172.21.15.138:0/35189681 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64ec03cea0 0x7f64ec03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:06.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.838+0000 7f64fffff640 10 monclient: get_auth_request con 0x7f64ec03cea0 auth_method 0 2024-09-11T05:14:06.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.842+0000 7f64fffff640 1 --2- 172.21.15.138:0/35189681 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64ec03cea0 0x7f64ec03f360 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 crypto rx=0x7f65000746c0 tx=0x7f64f800b3a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:06.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:06.842+0000 7f64e7fff640 10 client.22664.objecter ms_handle_connect 0x7f64ec03cea0 2024-09-11T05:14:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 10 monclient: get_auth_request con 0x7f6500134cb0 auth_method 0 2024-09-11T05:14:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:14:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:14:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.042+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:14:07.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:07.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 10 monclient: get_auth_request con 0x7f6500134cb0 auth_method 0 2024-09-11T05:14:07.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:07.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:07.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:07.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:14:07.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:14:07.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:07.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.446+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:14:07.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.666+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(314..314 src has 1..314) ==== 5361+0+0 (secure 0 0 0) 0x7f64f00e9df0 con 0x7f650009da20 2024-09-11T05:14:07.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.666+0000 7f64e7fff640 10 client.22664.objecter ms_dispatch 0x7f6500000ba0 osd_map(314..314 src has 1..314) 2024-09-11T05:14:07.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.666+0000 7f64e7fff640 3 client.22664.objecter handle_osd_map got epochs [314,314] > 0 2024-09-11T05:14:07.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.666+0000 7f64e7fff640 3 client.22664.objecter handle_osd_map decoding full epoch 314 2024-09-11T05:14:07.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:07.666+0000 7f64e7fff640 20 client.22664.objecter dump_active .. 0 homeless 2024-09-11T05:14:08.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:08.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 10 monclient: get_auth_request con 0x7f6500134cb0 auth_method 0 2024-09-11T05:14:08.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:08.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:08.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:08.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:08.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:08.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:08.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 10 monclient(hunting): handle_auth_done global_id 74098 payload 824 2024-09-11T05:14:08.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.250+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65000a26c0 con 0x7f6500134cb0 2024-09-11T05:14:08.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:08.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f64f4007fd0 tx=0x7f64f40136a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:08.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 10 client.22664.objecter ms_handle_connect 0x7f6500134cb0 2024-09-11T05:14:08.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 10 client.22664.objecter resend_mon_ops 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4017430 con 0x7f6500134cb0 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 10 monclient: dump: 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:08.257 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f64f4017430 con 0x7f6500134cb0 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f64f4007fd0 tx=0x7f64f40136a0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 msgr2=0x7f6500072760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:08.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.254+0000 7f64e7fff640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500134cb0 0x7f6500072760 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f64f4007fd0 tx=0x7f64f40136a0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f6505f50640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:14:08.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f6505f50640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500072ca0 0x7f65001387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:08.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f6505f50640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:08.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f65000730b0 con 0x7f6500072ca0 2024-09-11T05:14:08.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500072ca0 0x7f65001387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:08.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 10 monclient: get_auth_request con 0x7f6500072ca0 auth_method 0 2024-09-11T05:14:08.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:08.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:08.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:08.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:08.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:08.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:08.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 10 monclient(hunting): handle_auth_done global_id 74101 payload 824 2024-09-11T05:14:08.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.262+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f650011dc80 con 0x7f6500072ca0 2024-09-11T05:14:08.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64feffd640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500072ca0 msgr2=0x7f65001387c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 10 client.22664.objecter ms_handle_connect 0x7f6500072ca0 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 10 client.22664.objecter resend_mon_ops 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64feffd640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500072ca0 0x7f65001387c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005630 tx=0x7f64f4003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4026040 con 0x7f6500072ca0 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 10 monclient: dump: 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:08.270 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.271 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.271 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:08.271 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:08.271 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:08.271 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:08.271 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:08.271 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f64f4021a50 con 0x7f6500072ca0 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500072ca0 msgr2=0x7f65001387c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500072ca0 0x7f65001387c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005630 tx=0x7f64f4003700 comp rx=0 tx=0).stop 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500072ca0 msgr2=0x7f65001387c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.266+0000 7f64e7fff640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6500072ca0 0x7f65001387c0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005630 tx=0x7f64f4003700 comp rx=0 tx=0).stop 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:14:08.272 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 26, 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 15815, 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 5, 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:14:08.273 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:14:08.274 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.275 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:14:08.276 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:08.277 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:08.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.279 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:08.280 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:08.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:14:08.282 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:14:08.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:08.284 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:08.284 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:08.284 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:08.284 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:08.284 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:08.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:08.284 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:14:08.284 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:14:08.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 10 librados: watch_flush enter 2024-09-11T05:14:08.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 10 librados: watch_flush exit 2024-09-11T05:14:08.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 20 client.22664.objecter shutdown clearing up homeless session... 2024-09-11T05:14:08.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 10 client.22664.objecter successfully canceled tick 2024-09-11T05:14:08.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64ec03cea0 msgr2=0x7f64ec03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 --2- 172.21.15.138:0/35189681 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64ec03cea0 0x7f64ec03f360 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 crypto rx=0x7f65000746c0 tx=0x7f64f800b3a0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 10 monclient: shutdown 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f650009da20 msgr2=0x7f65001cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 --2- 172.21.15.138:0/35189681 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f650009da20 0x7f65001cbdc0 secure :-1 s=READY pgs=5776 cs=0 l=1 rev1=1 crypto rx=0x7f64f00af040 tx=0x7f64f000a870 comp rx=0 tx=0).stop 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f64fffff640 1 -- 172.21.15.138:0/35189681 reap_dead start 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f64fffff640 1 -- 172.21.15.138:0/35189681 reap_dead start 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 shutdown_connections 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 >> 172.21.15.138:0/35189681 conn(0x7f6500063640 msgr2=0x7f650013a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 shutdown_connections 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 -- 172.21.15.138:0/35189681 wait complete. 2024-09-11T05:14:08.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.270+0000 7f6505f50640 1 librados: shutdown 2024-09-11T05:14:08.294 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:14:08.294 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:14:08.294 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.414+0000 7f7478bb4640 10 monclient: get_monmap_and_config 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.414+0000 7f7478bb4640 10 monclient: build_initial_monmap 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.414+0000 7f7478bb4640 10 monclient: monmap: 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:08.420692+0000 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:08.420692+0000 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:08.419 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:08.420 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:08.420 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:08.420 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:08.420 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 1 Processor -- start 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 1 -- start start 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient: init 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f747406e0e0 0x7f7474136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient(hunting): picked mon.noname-c con 0x7f747406e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f7474136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient(hunting): picked mon.noname-b con 0x7f7474136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:08.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f74740a1920 addr v1:172.21.15.138:6789/0 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74740a3270 con 0x7f74740a1920 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f747406e4c0 con 0x7f747406e0e0 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f74740a0af0 con 0x7f7474136170 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7478bb4640 10 monclient(hunting): authenticate will time out at 1331.856974s 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7472d9d640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f74740a1920 0x7f74740a1db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:41444/0 (socket says 172.21.15.138:41444) 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7472d9d640 1 -- 172.21.15.138:0/253787891 learned_addr learned my addr 172.21.15.138:0/253787891 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 1 --2- 172.21.15.138:0/253787891 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f7474136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient(hunting): get_auth_request con 0x7f7474136170 auth_method 0 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:08.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f747259c640 1 --2- 172.21.15.138:0/253787891 >> v2:172.21.15.190:3300/0 conn(0x7f747406e0e0 0x7f7474136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f747259c640 10 monclient(hunting): get_auth_request con 0x7f747406e0e0 auth_method 0 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f747259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f747259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f747259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f747259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f747259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f747259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient(hunting): handle_auth_done global_id 22667 payload 274 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 1 -- 172.21.15.138:0/253787891 >> v2:172.21.15.190:3300/0 conn(0x7f747406e0e0 msgr2=0x7f7474136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 1 --2- 172.21.15.138:0/253787891 >> v2:172.21.15.190:3300/0 conn(0x7f747406e0e0 0x7f7474136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 1 -- 172.21.15.138:0/253787891 >> v1:172.21.15.138:6789/0 conn(0x7f74740a1920 legacy=0x7f74740a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:08.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:14:08.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 1 monclient: found mon.noname-b 2024-09-11T05:14:08.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:08.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:14:08.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.418+0000 7f7471d9b640 1 -- 172.21.15.138:0/253787891 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74740a2f90 con 0x7f7474136170 2024-09-11T05:14:08.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7471d9b640 1 -- 172.21.15.138:0/253787891 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 msgr2=0x7f7474136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:08.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7471d9b640 1 --2- 172.21.15.138:0/253787891 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f7474136c60 secure :-1 s=READY pgs=5780 cs=0 l=1 rev1=1 crypto rx=0x7f74680097c0 tx=0x7f746801c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b1edbe601749cbb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:08.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 1 -- 172.21.15.138:0/253787891 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74680b1070 con 0x7f7474136170 2024-09-11T05:14:08.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:08.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: dump: 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: _finish_auth 0 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: _check_auth_tickets 2024-09-11T05:14:08.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 1 -- 172.21.15.138:0/253787891 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74680ac440 con 0x7f7474136170 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 1 -- 172.21.15.138:0/253787891 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74680ac810 con 0x7f7474136170 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 5 monclient: authenticate success, global_id 22667 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 4 set_mon_vals no callback set 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f747159a640 10 monclient: dump: 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:08.428 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:08.429 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:08.429 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:08.429 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:08.429 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:08.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 10 monclient: shutdown 2024-09-11T05:14:08.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:08.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 1 -- 172.21.15.138:0/253787891 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 msgr2=0x7f7474136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 1 --2- 172.21.15.138:0/253787891 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f7474136c60 secure :-1 s=READY pgs=5780 cs=0 l=1 rev1=1 crypto rx=0x7f74680097c0 tx=0x7f746801c180 comp rx=0 tx=0).stop 2024-09-11T05:14:08.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 1 -- 172.21.15.138:0/253787891 shutdown_connections 2024-09-11T05:14:08.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 1 --2- 172.21.15.138:0/253787891 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f7474136c60 unknown :-1 s=CLOSED pgs=5780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 1 --2- 172.21.15.138:0/253787891 >> v2:172.21.15.190:3300/0 conn(0x7f747406e0e0 0x7f7474136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 1 -- 172.21.15.138:0/253787891 >> 172.21.15.138:0/253787891 conn(0x7f7474063640 msgr2=0x7f7474063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 1 -- 172.21.15.138:0/253787891 shutdown_connections 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.422+0000 7f7478bb4640 1 -- 172.21.15.138:0/253787891 wait complete. 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient: build_initial_monmap 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient: monmap: 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:08.433 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 librados: starting msgr at 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 librados: starting objecter 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 Processor -- start 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 -- start start 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 librados: setting wanted keys 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 librados: calling monclient init 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient: init 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f747406e0e0 0x7f74741ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient(hunting): picked mon.noname-b con 0x7f747406e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74740a1920 0x7f747413c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient(hunting): picked mon.noname-a con 0x7f74740a1920 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:08.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f747413cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient(hunting): picked mon.noname-c con 0x7f7474136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f74741410a0 con 0x7f74740a1920 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7474140f20 con 0x7f747406e0e0 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7474141220 con 0x7f7474136170 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7478bb4640 10 monclient(hunting): authenticate will time out at 1331.865218s 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74740a1920 0x7f747413c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f747413cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74740a1920 0x7f747413c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40364/0 (socket says 172.21.15.138:40364) 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 1 -- 172.21.15.138:0/746905710 learned_addr learned my addr 172.21.15.138:0/746905710 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:08.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 10 monclient(hunting): get_auth_request con 0x7f74740a1920 auth_method 0 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient(hunting): get_auth_request con 0x7f7474136170 auth_method 0 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f747259c640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f747406e0e0 0x7f74741ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:08.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient(hunting): handle_auth_done global_id 22670 payload 824 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 1 -- 172.21.15.138:0/746905710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f747406e0e0 msgr2=0x7f74741ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f747406e0e0 0x7f74741ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 1 -- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74740a1920 msgr2=0x7f747413c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74740a1920 0x7f747413c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 1 monclient: found mon.noname-c 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7472d9d640 1 -- 172.21.15.138:0/746905710 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747409e110 con 0x7f7474136170 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f7471d9b640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74740a1920 0x7f747413c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f747259c640 10 monclient: get_auth_request con 0x7f747406e0e0 auth_method 0 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.426+0000 7f747259c640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f747406e0e0 0x7f74741ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:14:08.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7472d9d640 1 -- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 msgr2=0x7f747413cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7472d9d640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f747413cf30 secure :-1 s=READY pgs=5781 cs=0 l=1 rev1=1 crypto rx=0x7f746c00c440 tx=0x7f746c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 client.?.objecter ms_handle_connect 0x7f7474136170 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 1 -- 172.21.15.138:0/746905710 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c017070 con 0x7f7474136170 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: dump: 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:08.438 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: _finish_auth 0 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 5 monclient: authenticate success, global_id 22670 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 1 -- 172.21.15.138:0/746905710 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f746c0070d0 con 0x7f7474136170 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 1 -- 172.21.15.138:0/746905710 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c0074c0 con 0x7f7474136170 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7463fff640 4 set_mon_vals no callback set 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 monclient: dump: 2024-09-11T05:14:08.439 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 10 monclient: _renew_subs 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7474138150 con 0x7f7474136170 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 10 client.22670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:08.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 10 monclient: _renew_subs 2024-09-11T05:14:08.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:08.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f747413a4b0 con 0x7f7474136170 2024-09-11T05:14:08.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 1 librados: init done 2024-09-11T05:14:08.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:08.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:08.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:08.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74740a1db0 con 0x7f7474136170 2024-09-11T05:14:08.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 1 -- 172.21.15.138:0/746905710 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f746c01b3b0 con 0x7f7474136170 2024-09-11T05:14:08.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 1 --2- 172.21.15.138:0/746905710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f745403cf20 0x7f745403f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 1 -- 172.21.15.138:0/746905710 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(315..315 src has 1..315) ==== 5374+0+0 (secure 0 0 0) 0x7f746c04f020 con 0x7f7474136170 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 10 client.22670.objecter ms_dispatch 0x7f7474000ba0 osd_map(315..315 src has 1..315) 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 3 client.22670.objecter handle_osd_map got epochs [315,315] > 0 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 3 client.22670.objecter handle_osd_map decoding full epoch 315 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f7462ffd640 20 client.22670.objecter dump_active .. 0 homeless 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f747259c640 1 --2- 172.21.15.138:0/746905710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f745403cf20 0x7f745403f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.430+0000 7f747259c640 10 monclient: get_auth_request con 0x7f745403cf20 auth_method 0 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.434+0000 7f7462ffd640 10 client.22670.objecter ms_handle_connect 0x7f745403cf20 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.434+0000 7f747259c640 1 --2- 172.21.15.138:0/746905710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f745403cf20 0x7f745403f3e0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7f7474137c80 tx=0x7f746400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.434+0000 7f7462ffd640 1 -- 172.21.15.138:0/746905710 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f746c01be80 con 0x7f7474136170 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.434+0000 7f7462ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:08.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.434+0000 7f7462ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:08.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.678+0000 7f7478bb4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:14:08.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.678+0000 7f7478bb4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:14:08.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.678+0000 7f7478bb4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:08.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.678+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f747406ccd0 con 0x7f7474136170 2024-09-11T05:14:08.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f7462ffd640 1 -- 172.21.15.138:0/746905710 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f746c01a360 con 0x7f7474136170 2024-09-11T05:14:08.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f7462ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:14:08.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f7462ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:08.686 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.686 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=10,logm=63} crc {auth=2138568001,config=692619927,health=2767134856,kv=310889196,logm=3013225887}) 2024-09-11T05:14:08.686 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2583320509,config=692619927,health=2767134856,kv=310889196,logm=2953679696}) 2024-09-11T05:14:08.686 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.686 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=12,kv=10,logm=63} crc {auth=2138568001,config=692619927,health=2767134856,kv=310889196,logm=3013225887}) 2024-09-11T05:14:08.686 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.682+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2583320509,config=692619927,health=2767134856,kv=310889196,logm=968433026}) 2024-09-11T05:14:08.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 10 librados: watch_flush enter 2024-09-11T05:14:08.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 10 librados: watch_flush exit 2024-09-11T05:14:08.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 20 client.22670.objecter shutdown clearing up homeless session... 2024-09-11T05:14:08.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 10 client.22670.objecter successfully canceled tick 2024-09-11T05:14:08.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f745403cf20 msgr2=0x7f745403f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 --2- 172.21.15.138:0/746905710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f745403cf20 0x7f745403f3e0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7f7474137c80 tx=0x7f746400b040 comp rx=0 tx=0).stop 2024-09-11T05:14:08.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 10 monclient: shutdown 2024-09-11T05:14:08.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:08.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 msgr2=0x7f747413cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:08.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f747413cf30 secure :-1 s=READY pgs=5781 cs=0 l=1 rev1=1 crypto rx=0x7f746c00c440 tx=0x7f746c00c900 comp rx=0 tx=0).stop 2024-09-11T05:14:08.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 shutdown_connections 2024-09-11T05:14:08.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 --2- 172.21.15.138:0/746905710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f745403cf20 0x7f745403f3e0 unknown :-1 s=CLOSED pgs=3402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7474136170 0x7f747413cf30 unknown :-1 s=CLOSED pgs=5781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f74740a1920 0x7f747413c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 --2- 172.21.15.138:0/746905710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f747406e0e0 0x7f74741ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:08.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 >> 172.21.15.138:0/746905710 conn(0x7f7474063640 msgr2=0x7f74740a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:08.691 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.691 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166734653}) 2024-09-11T05:14:08.691 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521580774}) 2024-09-11T05:14:08.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 shutdown_connections 2024-09-11T05:14:08.691 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 -- 172.21.15.138:0/746905710 wait complete. 2024-09-11T05:14:08.692 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166734653}) 2024-09-11T05:14:08.692 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240444967}) 2024-09-11T05:14:08.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f7478bb4640 1 librados: shutdown 2024-09-11T05:14:08.693 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.693 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223115763}) 2024-09-11T05:14:08.693 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680135644}) 2024-09-11T05:14:08.694 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.694 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223115763}) 2024-09-11T05:14:08.694 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.686+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157152400}) 2024-09-11T05:14:08.705 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:14:08.725 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.722+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.725 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.722+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918694168}) 2024-09-11T05:14:08.725 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.722+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245928900}) 2024-09-11T05:14:08.725 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.722+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.725 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.722+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918694168}) 2024-09-11T05:14:08.726 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.722+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3733677353}) 2024-09-11T05:14:08.733 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.730+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.730+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670651135}) 2024-09-11T05:14:08.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.730+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372686477}) 2024-09-11T05:14:08.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.730+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.730+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670651135}) 2024-09-11T05:14:08.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.730+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031450401}) 2024-09-11T05:14:08.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.898+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.898+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329657640}) 2024-09-11T05:14:08.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.898+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1439098919}) 2024-09-11T05:14:08.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.898+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:08.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.898+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329657640}) 2024-09-11T05:14:08.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:08.898+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871477369}) 2024-09-11T05:14:09.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701647831}) 2024-09-11T05:14:09.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3966613800}) 2024-09-11T05:14:09.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701647831}) 2024-09-11T05:14:09.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967566007}) 2024-09-11T05:14:09.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7383582}) 2024-09-11T05:14:09.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1736173862}) 2024-09-11T05:14:09.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7383582}) 2024-09-11T05:14:09.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.166+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844857616}) 2024-09-11T05:14:09.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.170+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.170+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932377795}) 2024-09-11T05:14:09.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.170+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680697517}) 2024-09-11T05:14:09.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.170+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.170+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932377795}) 2024-09-11T05:14:09.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.170+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919539397}) 2024-09-11T05:14:09.176 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.177 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=336136241}) 2024-09-11T05:14:09.177 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1522297845}) 2024-09-11T05:14:09.177 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.177 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=336136241}) 2024-09-11T05:14:09.177 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1168759256}) 2024-09-11T05:14:09.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428562903}) 2024-09-11T05:14:09.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=647071994}) 2024-09-11T05:14:09.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428562903}) 2024-09-11T05:14:09.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.174+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944744798}) 2024-09-11T05:14:09.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987263011}) 2024-09-11T05:14:09.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1279750693}) 2024-09-11T05:14:09.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987263011}) 2024-09-11T05:14:09.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1013030679}) 2024-09-11T05:14:09.184 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716246295}) 2024-09-11T05:14:09.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2958100054}) 2024-09-11T05:14:09.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716246295}) 2024-09-11T05:14:09.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3898913170}) 2024-09-11T05:14:09.187 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.187 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438220010}) 2024-09-11T05:14:09.187 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528131936}) 2024-09-11T05:14:09.187 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.187 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438220010}) 2024-09-11T05:14:09.187 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749849798}) 2024-09-11T05:14:09.189 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.186+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.190 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.186+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077122967}) 2024-09-11T05:14:09.190 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.186+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=771559574}) 2024-09-11T05:14:09.190 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.186+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.190 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.186+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077122967}) 2024-09-11T05:14:09.190 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.186+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105182963}) 2024-09-11T05:14:09.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698735899}) 2024-09-11T05:14:09.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153338776}) 2024-09-11T05:14:09.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.193 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698735899}) 2024-09-11T05:14:09.193 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2292559079}) 2024-09-11T05:14:09.195 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.195 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157060559}) 2024-09-11T05:14:09.195 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258683163}) 2024-09-11T05:14:09.195 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.195 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157060559}) 2024-09-11T05:14:09.195 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796189293}) 2024-09-11T05:14:09.198 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.194+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.198 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.194+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871909451}) 2024-09-11T05:14:09.198 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.194+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940484902}) 2024-09-11T05:14:09.198 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.194+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.198 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.194+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3871909451}) 2024-09-11T05:14:09.198 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.194+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=799325845}) 2024-09-11T05:14:09.200 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.200 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2133495517}) 2024-09-11T05:14:09.200 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=232850409}) 2024-09-11T05:14:09.201 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.201 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2133495517}) 2024-09-11T05:14:09.201 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2099564507}) 2024-09-11T05:14:09.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809294520}) 2024-09-11T05:14:09.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718873830}) 2024-09-11T05:14:09.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809294520}) 2024-09-11T05:14:09.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140391545}) 2024-09-11T05:14:09.205 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.202+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.206 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.202+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4238170690}) 2024-09-11T05:14:09.206 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.202+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=281400908}) 2024-09-11T05:14:09.206 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.202+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.206 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.202+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4238170690}) 2024-09-11T05:14:09.206 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.202+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1419154290}) 2024-09-11T05:14:09.208 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.208 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921777357}) 2024-09-11T05:14:09.208 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2576131158}) 2024-09-11T05:14:09.208 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.208 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921777357}) 2024-09-11T05:14:09.208 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2114705989}) 2024-09-11T05:14:09.210 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.210 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266548750}) 2024-09-11T05:14:09.211 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624886416}) 2024-09-11T05:14:09.211 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.211 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=266548750}) 2024-09-11T05:14:09.211 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.206+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618574259}) 2024-09-11T05:14:09.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=489965956}) 2024-09-11T05:14:09.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1290246902}) 2024-09-11T05:14:09.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=489965956}) 2024-09-11T05:14:09.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=763173085}) 2024-09-11T05:14:09.215 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.215 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680310190}) 2024-09-11T05:14:09.215 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.210+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3780681270}) 2024-09-11T05:14:09.216 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.216 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680310190}) 2024-09-11T05:14:09.216 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039922917}) 2024-09-11T05:14:09.218 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.218 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1958614899}) 2024-09-11T05:14:09.218 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162616550}) 2024-09-11T05:14:09.218 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.218 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1958614899}) 2024-09-11T05:14:09.218 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524051665}) 2024-09-11T05:14:09.220 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.221 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517512776}) 2024-09-11T05:14:09.221 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465188269}) 2024-09-11T05:14:09.221 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.221 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517512776}) 2024-09-11T05:14:09.221 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671448265}) 2024-09-11T05:14:09.223 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.223 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415875757}) 2024-09-11T05:14:09.223 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1050676198}) 2024-09-11T05:14:09.223 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.223 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415875757}) 2024-09-11T05:14:09.223 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.218+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1454457794}) 2024-09-11T05:14:09.226 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.222+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.226 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.222+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240901154}) 2024-09-11T05:14:09.226 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.222+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=762782239}) 2024-09-11T05:14:09.226 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.222+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.226 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.222+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240901154}) 2024-09-11T05:14:09.226 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.222+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935846742}) 2024-09-11T05:14:09.228 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.228 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901248701}) 2024-09-11T05:14:09.228 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076005630}) 2024-09-11T05:14:09.228 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.229 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901248701}) 2024-09-11T05:14:09.229 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2005322945}) 2024-09-11T05:14:09.231 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.231 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573193657}) 2024-09-11T05:14:09.231 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3412007859}) 2024-09-11T05:14:09.231 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.231 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573193657}) 2024-09-11T05:14:09.231 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.226+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1691636629}) 2024-09-11T05:14:09.234 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.230+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.234 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.230+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13} crc {logm=4067347578,mdsmap=1754867906,mgr=2081689912,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=847705869,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=197690052,osd_snap=3916499506}) 2024-09-11T05:14:09.234 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.230+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=53032143,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=1369135550,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=197690052,osd_snap=2211640305}) 2024-09-11T05:14:09.234 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.230+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.234 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.230+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13} crc {logm=4067347578,mdsmap=1754867906,mgr=2081689912,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=847705869,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=197690052,osd_snap=3916499506}) 2024-09-11T05:14:09.234 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.230+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=43,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=3478927175,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=1541506574,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=197690052,osd_snap=895377369}) 2024-09-11T05:14:09.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.234+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.234+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=4276467601,osdmap=4278444350}) 2024-09-11T05:14:09.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.234+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=4276467601,osdmap=1022002231}) 2024-09-11T05:14:09.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.234+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.234+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=4276467601,osdmap=4278444350}) 2024-09-11T05:14:09.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.234+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=4276467601,osdmap=1941256340}) 2024-09-11T05:14:09.240 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.240 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=104552487}) 2024-09-11T05:14:09.240 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3458910805}) 2024-09-11T05:14:09.240 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.240 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=104552487}) 2024-09-11T05:14:09.240 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2663269054}) 2024-09-11T05:14:09.243 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.243 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=648197452}) 2024-09-11T05:14:09.243 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1931148499}) 2024-09-11T05:14:09.243 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.243 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=648197452}) 2024-09-11T05:14:09.243 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.238+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1786665850}) 2024-09-11T05:14:09.248 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.246+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.248 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.246+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=681802795}) 2024-09-11T05:14:09.248 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.246+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2956001364}) 2024-09-11T05:14:09.248 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.246+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.248 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.246+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=681802795}) 2024-09-11T05:14:09.248 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.246+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3397804517}) 2024-09-11T05:14:09.262 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.258+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.262 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.258+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=3727139429}) 2024-09-11T05:14:09.263 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.258+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=4210350622}) 2024-09-11T05:14:09.263 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.258+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:14:09.263 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.258+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=3727139429}) 2024-09-11T05:14:09.263 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:14:09.258+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=77} crc {osdmap=208555393}) 2024-09-11T05:14:09.706 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:14:09.706 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient: get_monmap_and_config 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient: build_initial_monmap 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient: monmap: 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:09.838678+0000 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:09.838678+0000 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:09.837 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:09.838 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:09.838 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:09.838 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:09.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 1 Processor -- start 2024-09-11T05:14:09.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 1 -- start start 2024-09-11T05:14:09.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient: init 2024-09-11T05:14:09.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:09.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-11T05:14:09.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7b00a3940 addr v1:172.21.15.138:6789/0 2024-09-11T05:14:09.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff7b006ce20 0x7ff7b013da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:09.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient(hunting): picked mon.noname-c con 0x7ff7b006ce20 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff7b00a4430 0x7ff7b013ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient(hunting): picked mon.noname-a con 0x7ff7b00a4430 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7b009cc20 con 0x7ff7b00a3940 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7ff7b009ded0 con 0x7ff7b00a4430 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff7b009d550 con 0x7ff7b006ce20 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b6ca2640 10 monclient(hunting): authenticate will time out at 1333.275560s 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b523f640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff7b00a4430 0x7ff7b013ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b523f640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff7b00a4430 0x7ff7b013ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40378/0 (socket says 172.21.15.138:40378) 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b523f640 1 -- 172.21.15.138:0/3427902475 learned_addr learned my addr 172.21.15.138:0/3427902475 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b523f640 10 monclient(hunting): get_auth_request con 0x7ff7b00a4430 auth_method 0 2024-09-11T05:14:09.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b523f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b523f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.834+0000 7ff7b523f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 1 --2- 172.21.15.138:0/3427902475 >> v2:172.21.15.190:3300/0 conn(0x7ff7b006ce20 0x7ff7b013da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient(hunting): get_auth_request con 0x7ff7b006ce20 auth_method 0 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient(hunting): handle_auth_done global_id 74107 payload 274 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 1 -- 172.21.15.138:0/3427902475 >> v2:172.21.15.138:3300/0 conn(0x7ff7b00a4430 msgr2=0x7ff7b013ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 1 --2- 172.21.15.138:0/3427902475 >> v2:172.21.15.138:3300/0 conn(0x7ff7b00a4430 0x7ff7b013ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:09.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 1 -- 172.21.15.138:0/3427902475 >> v1:172.21.15.138:6789/0 conn(0x7ff7b00a3940 legacy=0x7ff7b00a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:09.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient: _finish_hunting 0 2024-09-11T05:14:09.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 1 monclient: found mon.noname-c 2024-09-11T05:14:09.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:09.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:14:09.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.838+0000 7ff7affff640 1 -- 172.21.15.138:0/3427902475 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7b009cf00 con 0x7ff7b006ce20 2024-09-11T05:14:09.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7affff640 1 -- 172.21.15.138:0/3427902475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 msgr2=0x7ff7b013da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:09.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7affff640 1 --2- 172.21.15.138:0/3427902475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 0x7ff7b013da40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff7b00a57d0 tx=0x7ff7a001c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e43f92bb0279d751 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:09.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 1 -- 172.21.15.138:0/3427902475 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00b0070 con 0x7ff7b006ce20 2024-09-11T05:14:09.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: dump: 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: _finish_auth 0 2024-09-11T05:14:09.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 1 -- 172.21.15.138:0/3427902475 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7a00ab600 con 0x7ff7b006ce20 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 1 -- 172.21.15.138:0/3427902475 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00ab9d0 con 0x7ff7b006ce20 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7af7fe640 10 monclient: dump: 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:09.847 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:09.848 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:09.848 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:09.848 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:09.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 5 monclient: authenticate success, global_id 74107 2024-09-11T05:14:09.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:09.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 4 set_mon_vals no callback set 2024-09-11T05:14:09.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 10 monclient: shutdown 2024-09-11T05:14:09.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:09.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/3427902475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 msgr2=0x7ff7b013da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:09.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/3427902475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 0x7ff7b013da40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff7b00a57d0 tx=0x7ff7a001c3a0 comp rx=0 tx=0).stop 2024-09-11T05:14:09.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/3427902475 shutdown_connections 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/3427902475 >> v2:172.21.15.138:3300/0 conn(0x7ff7b00a4430 0x7ff7b013ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/3427902475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 0x7ff7b013da40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/3427902475 >> 172.21.15.138:0/3427902475 conn(0x7ff7b00a8530 msgr2=0x7ff7b00a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/3427902475 shutdown_connections 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/3427902475 wait complete. 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 10 monclient: build_initial_monmap 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.842+0000 7ff7b6ca2640 10 monclient: monmap: 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:09.854 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 librados: starting msgr at 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 librados: starting objecter 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 Processor -- start 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 -- start start 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 librados: setting wanted keys 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 librados: calling monclient init 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient: init 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 0x7ff7b011f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient(hunting): picked mon.noname-b con 0x7ff7b006ce20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:09.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff7b00a3940 0x7ff7b0121980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient(hunting): picked mon.noname-a con 0x7ff7b00a3940 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff7b00a4430 0x7ff7b01387f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient(hunting): picked mon.noname-c con 0x7ff7b00a4430 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b0142e70 con 0x7ff7b00a3940 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b0142cf0 con 0x7ff7b006ce20 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff7b0142ff0 con 0x7ff7b00a4430 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b6ca2640 10 monclient(hunting): authenticate will time out at 1333.285057s 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff7b00a3940 0x7ff7b0121980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:09.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff7b00a3940 0x7ff7b0121980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40388/0 (socket says 172.21.15.138:40388) 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 -- 172.21.15.138:0/1592245309 learned_addr learned my addr 172.21.15.138:0/1592245309 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient(hunting): get_auth_request con 0x7ff7b00a3940 auth_method 0 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff7b00a4430 0x7ff7b01387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 10 monclient(hunting): get_auth_request con 0x7ff7b00a4430 auth_method 0 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:09.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b4a3e640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 0x7ff7b011f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b4a3e640 10 monclient(hunting): get_auth_request con 0x7ff7b006ce20 auth_method 0 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b4a3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b4a3e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b4a3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient(hunting): handle_auth_done global_id 64332 payload 824 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 -- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff7b00a4430 msgr2=0x7ff7b01387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff7b00a4430 0x7ff7b01387f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 -- 172.21.15.138:0/1592245309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 msgr2=0x7ff7b011f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 0x7ff7b011f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:09.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient: _finish_hunting 0 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 monclient: found mon.noname-a 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 -- 172.21.15.138:0/1592245309 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7b0139490 con 0x7ff7b00a3940 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7b523f640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff7b00a4430 0x7ff7b01387f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 -- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff7b00a3940 msgr2=0x7ff7b0121980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7affff640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff7b00a3940 0x7ff7b0121980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0019fd0 tx=0x7ff7a001abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7ad7fa640 10 client.?.objecter ms_handle_connect 0x7ff7b00a3940 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.846+0000 7ff7ad7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 1 -- 172.21.15.138:0/1592245309 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00b0050 con 0x7ff7b00a3940 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: dump: 2024-09-11T05:14:09.859 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: _finish_auth 0 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 1 -- 172.21.15.138:0/1592245309 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7a00bc070 con 0x7ff7b00a3940 2024-09-11T05:14:09.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 5 monclient: authenticate success, global_id 64332 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 10 monclient: _renew_subs 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7b009f120 con 0x7ff7b00a3940 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 1 -- 172.21.15.138:0/1592245309 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00b5a40 con 0x7ff7b00a3940 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 monclient: dump: 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:09.861 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7aeffd640 4 set_mon_vals no callback set 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 10 client.64332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 10 monclient: _renew_subs 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7b01228e0 con 0x7ff7b00a3940 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 1 librados: init done 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:09.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 1 -- 172.21.15.138:0/1592245309 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7ff7a00b5be0 con 0x7ff7b00a3940 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7b00a3d20 con 0x7ff7b00a3940 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 1 --2- 172.21.15.138:0/1592245309 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff79c03cea0 0x7ff79c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 1 -- 172.21.15.138:0/1592245309 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(317..317 src has 1..317) ==== 5361+0+0 (secure 0 0 0) 0x7ff7a00ea2b0 con 0x7ff7b00a3940 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 10 client.64332.objecter ms_dispatch 0x7ff7b0000ba0 osd_map(317..317 src has 1..317) 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 3 client.64332.objecter handle_osd_map got epochs [317,317] > 0 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 3 client.64332.objecter handle_osd_map decoding full epoch 317 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7ad7fa640 20 client.64332.objecter dump_active .. 0 homeless 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b4a3e640 1 --2- 172.21.15.138:0/1592245309 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff79c03cea0 0x7ff79c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:09.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.850+0000 7ff7b4a3e640 10 monclient: get_auth_request con 0x7ff79c03cea0 auth_method 0 2024-09-11T05:14:09.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.854+0000 7ff7b4a3e640 1 --2- 172.21.15.138:0/1592245309 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff79c03cea0 0x7ff79c03f360 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7ff7b01230b0 tx=0x7ff7a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:09.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.854+0000 7ff7ad7fa640 10 client.64332.objecter ms_handle_connect 0x7ff79c03cea0 2024-09-11T05:14:09.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.854+0000 7ff7ad7fa640 1 -- 172.21.15.138:0/1592245309 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7ff7a0016050 con 0x7ff7b00a3940 2024-09-11T05:14:09.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.854+0000 7ff7ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:09.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:09.854+0000 7ff7ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:10.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.094+0000 7ff7b6ca2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:14:10.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.094+0000 7ff7b6ca2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:10.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.094+0000 7ff7b6ca2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.094+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff7b0122ca0 con 0x7ff7b00a3940 2024-09-11T05:14:10.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.094+0000 7ff7ad7fa640 1 -- 172.21.15.138:0/1592245309 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7ff7a00e8a60 con 0x7ff7b00a3940 2024-09-11T05:14:10.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.094+0000 7ff7ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:10.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.094+0000 7ff7ad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:10.097 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:14:10.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 10 librados: watch_flush enter 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 10 librados: watch_flush exit 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 20 client.64332.objecter shutdown clearing up homeless session... 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 10 client.64332.objecter successfully canceled tick 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff79c03cea0 msgr2=0x7ff79c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/1592245309 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff79c03cea0 0x7ff79c03f360 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7ff7b01230b0 tx=0x7ff7a4009040 comp rx=0 tx=0).stop 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 10 monclient: shutdown 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff7b00a3940 msgr2=0x7ff7b0121980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff7b00a3940 0x7ff7b0121980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0019fd0 tx=0x7ff7a001abe0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 shutdown_connections 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/1592245309 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff79c03cea0 0x7ff79c03f360 unknown :-1 s=CLOSED pgs=3403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff7b00a4430 0x7ff7b01387f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff7b00a3940 0x7ff7b0121980 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 --2- 172.21.15.138:0/1592245309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7b006ce20 0x7ff7b011f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 >> 172.21.15.138:0/1592245309 conn(0x7ff7b00a8530 msgr2=0x7ff7b00a1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:10.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 shutdown_connections 2024-09-11T05:14:10.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 -- 172.21.15.138:0/1592245309 wait complete. 2024-09-11T05:14:10.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.098+0000 7ff7b6ca2640 1 librados: shutdown 2024-09-11T05:14:10.117 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:14:10.117 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:14:10.117 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:14:10.117 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:14:10.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient: get_monmap_and_config 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient: build_initial_monmap 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient: monmap: 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:10.241565+0000 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:10.241565+0000 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:10.240 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:10.241 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:10.241 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:10.241 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:10.241 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 1 Processor -- start 2024-09-11T05:14:10.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 1 -- start start 2024-09-11T05:14:10.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient: init 2024-09-11T05:14:10.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feffc070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feffc09e6e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7feffc09eb10 0x7feffc140d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient(hunting): picked mon.noname-a con 0x7feffc09eb10 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc135b60 con 0x7feffc09e6e0 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc0a8990 con 0x7feffc070010 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7feffc0703f0 con 0x7feffc09eb10 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff003fbd640 10 monclient(hunting): authenticate will time out at 1333.677761s 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7feffc09eb10 0x7feffc140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7feffc09eb10 0x7feffc140d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40394/0 (socket says 172.21.15.138:40394) 2024-09-11T05:14:10.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 1 -- 172.21.15.138:0/1846286563 learned_addr learned my addr 172.21.15.138:0/1846286563 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:10.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 10 monclient(hunting): get_auth_request con 0x7feffc09eb10 auth_method 0 2024-09-11T05:14:10.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.238+0000 7ff00255a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff001558640 1 -- 172.21.15.138:0/1846286563 >> v1:172.21.15.138:6789/0 conn(0x7feffc09e6e0 legacy=0x7feffc13d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff00255a640 10 monclient(hunting): handle_auth_done global_id 64335 payload 274 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff00255a640 1 -- 172.21.15.138:0/1846286563 >> v1:172.21.15.190:6789/0 conn(0x7feffc070010 legacy=0x7feffc135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff00255a640 1 -- 172.21.15.138:0/1846286563 >> v1:172.21.15.138:6789/0 conn(0x7feffc09e6e0 legacy=0x7feffc13d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff00255a640 10 monclient: _finish_hunting 0 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff00255a640 1 monclient: found mon.noname-a 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff00255a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff00255a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.242+0000 7ff00255a640 1 -- 172.21.15.138:0/1846286563 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc09cbf0 con 0x7feffc09eb10 2024-09-11T05:14:10.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 1 -- 172.21.15.138:0/1846286563 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc09eb10 msgr2=0x7feffc140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 1 --2- 172.21.15.138:0/1846286563 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc09eb10 0x7feffc140d70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feff40098f0 tx=0x7feff401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1e2dd4122f689c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 1 -- 172.21.15.138:0/1846286563 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40a7070 con 0x7feffc09eb10 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: dump: 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.249 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: _finish_auth 0 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: _check_auth_tickets 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 1 -- 172.21.15.138:0/1846286563 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff40b0070 con 0x7feffc09eb10 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 1 -- 172.21.15.138:0/1846286563 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40ab890 con 0x7feffc09eb10 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff000d57640 10 monclient: dump: 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.250 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.251 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.251 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.251 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.251 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.251 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 5 monclient: authenticate success, global_id 64335 2024-09-11T05:14:10.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:10.251 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 4 set_mon_vals no callback set 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient: shutdown 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- 172.21.15.138:0/1846286563 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc09eb10 msgr2=0x7feffc140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 --2- 172.21.15.138:0/1846286563 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc09eb10 0x7feffc140d70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feff40098f0 tx=0x7feff401c470 comp rx=0 tx=0).stop 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- 172.21.15.138:0/1846286563 shutdown_connections 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 --2- 172.21.15.138:0/1846286563 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc09eb10 0x7feffc140d70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- 172.21.15.138:0/1846286563 >> 172.21.15.138:0/1846286563 conn(0x7feffc0ab830 msgr2=0x7feffc0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- 172.21.15.138:0/1846286563 shutdown_connections 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- 172.21.15.138:0/1846286563 wait complete. 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient: build_initial_monmap 2024-09-11T05:14:10.257 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient: monmap: 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 librados: starting msgr at 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 librados: starting objecter 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 Processor -- start 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- start start 2024-09-11T05:14:10.258 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 librados: setting wanted keys 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 librados: calling monclient init 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient: init 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc070010 0x7feffc1d0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient(hunting): picked mon.noname-a con 0x7feffc070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc09e6e0 0x7feffc1402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient(hunting): picked mon.noname-b con 0x7feffc09e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7feffc09eb10 0x7feffc140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient(hunting): picked mon.noname-c con 0x7feffc09eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feffc143e60 con 0x7feffc070010 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feffc143ce0 con 0x7feffc09e6e0 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7feffc143fe0 con 0x7feffc09eb10 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff003fbd640 10 monclient(hunting): authenticate will time out at 1333.687224s 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc070010 0x7feffc1d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc070010 0x7feffc1d0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40400/0 (socket says 172.21.15.138:40400) 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 -- 172.21.15.138:0/2218693232 learned_addr learned my addr 172.21.15.138:0/2218693232 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient(hunting): get_auth_request con 0x7feffc070010 auth_method 0 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7feffc09eb10 0x7feffc140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 10 monclient(hunting): get_auth_request con 0x7feffc09eb10 auth_method 0 2024-09-11T05:14:10.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff00255a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001558640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc09e6e0 0x7feffc1402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient(hunting): handle_auth_done global_id 64341 payload 824 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7feffc09eb10 msgr2=0x7feffc140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7feffc09eb10 0x7feffc140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.261 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc09e6e0 msgr2=0x7feffc1402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc09e6e0 0x7feffc1402c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient: _finish_hunting 0 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 monclient: found mon.noname-a 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001d59640 1 -- 172.21.15.138:0/2218693232 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc13ad70 con 0x7feffc070010 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001558640 10 monclient: get_auth_request con 0x7feffc09e6e0 auth_method 0 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.246+0000 7ff001558640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feffc09e6e0 0x7feffc1402c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001d59640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc070010 msgr2=0x7feffc1d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001d59640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc070010 0x7feffc1d0210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feff800c7c0 tx=0x7feff800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 client.?.objecter ms_handle_connect 0x7feffc070010 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff8016020 con 0x7feffc070010 2024-09-11T05:14:10.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:10.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: dump: 2024-09-11T05:14:10.263 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.263 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.263 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.264 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.264 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: _finish_auth 0 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: _check_auth_tickets 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff800ac60 con 0x7feffc070010 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 5 monclient: authenticate success, global_id 64341 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 10 monclient: _renew_subs 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc1d0930 con 0x7feffc070010 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff8012660 con 0x7feffc070010 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 monclient: dump: 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.266 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 10 client.64341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 10 monclient: _renew_subs 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc140d40 con 0x7feffc070010 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff37fe640 4 set_mon_vals no callback set 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 1 librados: init done 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc136f50 0x7feffc137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feffc135450 con 0x7feffc136f50 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc136f50 0x7feffc137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 10 monclient: get_auth_request con 0x7feffc136f50 auth_method 0 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7feff80128c0 con 0x7feffc070010 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 10 monclient(hunting): handle_auth_done global_id 64344 payload 824 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff001558640 1 -- 172.21.15.138:0/2218693232 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feffc13a960 con 0x7feffc136f50 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 1 --2- 172.21.15.138:0/2218693232 >> v2:172.21.15.190:6800/1562782189 conn(0x7fefe003cef0 0x7fefe003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(317..317 src has 1..317) ==== 5361+0+0 (secure 0 0 0) 0x7feff8050060 con 0x7feffc070010 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 10 client.64341.objecter ms_dispatch 0x7feffc000ba0 osd_map(317..317 src has 1..317) 2024-09-11T05:14:10.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 3 client.64341.objecter handle_osd_map got epochs [317,317] > 0 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 3 client.64341.objecter handle_osd_map decoding full epoch 317 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7feff27fc640 20 client.64341.objecter dump_active .. 0 homeless 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff00255a640 1 --2- 172.21.15.138:0/2218693232 >> v2:172.21.15.190:6800/1562782189 conn(0x7fefe003cef0 0x7fefe003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.250+0000 7ff00255a640 10 monclient: get_auth_request con 0x7fefe003cef0 auth_method 0 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7ff001558640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc136f50 msgr2=0x7feffc137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7ff001558640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc136f50 0x7feffc137330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feffc1385c0 tx=0x7fefec00d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 10 client.64341.objecter ms_handle_connect 0x7feffc136f50 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 10 client.64341.objecter resend_mon_ops 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec018070 con 0x7feffc136f50 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 10 monclient: dump: 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.269 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.270 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.270 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.270 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.270 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.270 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.270 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.270 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.270 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fefec0057d0 con 0x7feffc136f50 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc136f50 msgr2=0x7feffc137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc136f50 0x7feffc137330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feffc1385c0 tx=0x7fefec00d950 comp rx=0 tx=0).stop 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc136f50 msgr2=0x7feffc137330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc136f50 0x7feffc137330 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feffc1385c0 tx=0x7fefec00d950 comp rx=0 tx=0).stop 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7feff27fc640 10 client.64341.objecter ms_handle_connect 0x7fefe003cef0 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.254+0000 7ff00255a640 1 --2- 172.21.15.138:0/2218693232 >> v2:172.21.15.190:6800/1562782189 conn(0x7fefe003cef0 0x7fefe003f3b0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7feffc06efb0 tx=0x7feff40a7000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc137870 0x7feffc0a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7feffc139570 con 0x7feffc137870 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc137870 0x7feffc0a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 10 monclient: get_auth_request con 0x7feffc137870 auth_method 0 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 10 monclient(hunting): handle_auth_done global_id 64347 payload 824 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 1 -- 172.21.15.138:0/2218693232 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feffc069fa0 con 0x7feffc137870 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc137870 msgr2=0x7feffc0a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 10 client.64341.objecter ms_handle_connect 0x7feffc137870 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 10 client.64341.objecter resend_mon_ops 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff001558640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc137870 0x7feffc0a24a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7feffc1397f0 tx=0x7fefec000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec018040 con 0x7feffc137870 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 10 monclient: dump: 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fefec01a910 con 0x7feffc137870 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc137870 msgr2=0x7feffc0a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc137870 0x7feffc0a24a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7feffc1397f0 tx=0x7fefec000ee0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc137870 msgr2=0x7feffc0a24a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7feff27fc640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc137870 0x7feffc0a24a0 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7feffc1397f0 tx=0x7fefec000ee0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:14:10.277 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 26, 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 153892, 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 7, 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.278 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.279 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:14:10.280 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.281 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:10.282 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.284 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.285 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.286 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:14:10.287 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.288 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:14:10.289 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 10 librados: watch_flush enter 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 10 librados: watch_flush exit 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 20 client.64341.objecter shutdown clearing up homeless session... 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 10 client.64341.objecter successfully canceled tick 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 >> v2:172.21.15.190:6800/1562782189 conn(0x7fefe003cef0 msgr2=0x7fefe003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 --2- 172.21.15.138:0/2218693232 >> v2:172.21.15.190:6800/1562782189 conn(0x7fefe003cef0 0x7fefe003f3b0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7feffc06efb0 tx=0x7feff40a7000 comp rx=0 tx=0).stop 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 10 monclient: shutdown 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc070010 msgr2=0x7feffc1d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 --2- 172.21.15.138:0/2218693232 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7feffc070010 0x7feffc1d0210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feff800c7c0 tx=0x7feff800cc80 comp rx=0 tx=0).stop 2024-09-11T05:14:10.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff00255a640 1 -- 172.21.15.138:0/2218693232 reap_dead start 2024-09-11T05:14:10.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff00255a640 1 -- 172.21.15.138:0/2218693232 reap_dead start 2024-09-11T05:14:10.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 shutdown_connections 2024-09-11T05:14:10.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 >> 172.21.15.138:0/2218693232 conn(0x7feffc0ab830 msgr2=0x7feffc0aa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:10.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 shutdown_connections 2024-09-11T05:14:10.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 -- 172.21.15.138:0/2218693232 wait complete. 2024-09-11T05:14:10.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.262+0000 7ff003fbd640 1 librados: shutdown 2024-09-11T05:14:10.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:14:10.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:14:10.293 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient: get_monmap_and_config 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient: build_initial_monmap 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient: monmap: 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:10.406675+0000 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:10.406675+0000 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:10.405 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:10.406 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:10.406 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:10.406 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 1 Processor -- start 2024-09-11T05:14:10.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 1 -- start start 2024-09-11T05:14:10.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient: init 2024-09-11T05:14:10.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:10.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-11T05:14:10.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38a0070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:10.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f38a009e6e0 0x7f38a013d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient(hunting): picked mon.noname-a con 0x7f38a009e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f38a009eb10 0x7f38a013fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient(hunting): picked mon.noname-c con 0x7f38a009eb10 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38a0135b60 con 0x7f38a0070010 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f38a00703f0 con 0x7f38a009e6e0 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f38a00a8990 con 0x7f38a009eb10 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a79d2640 10 monclient(hunting): authenticate will time out at 1333.842444s 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f38a009e6e0 0x7f38a013d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f38a009e6e0 0x7f38a013d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40428/0 (socket says 172.21.15.138:40428) 2024-09-11T05:14:10.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 1 -- 172.21.15.138:0/1714156966 learned_addr learned my addr 172.21.15.138:0/1714156966 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient(hunting): get_auth_request con 0x7f38a009e6e0 auth_method 0 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a5f6f640 1 --2- 172.21.15.138:0/1714156966 >> v2:172.21.15.190:3300/0 conn(0x7f38a009eb10 0x7f38a013fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_done global_id 64350 payload 274 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 1 -- 172.21.15.138:0/1714156966 >> v2:172.21.15.190:3300/0 conn(0x7f38a009eb10 msgr2=0x7f38a013fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 1 --2- 172.21.15.138:0/1714156966 >> v2:172.21.15.190:3300/0 conn(0x7f38a009eb10 0x7f38a013fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 1 -- 172.21.15.138:0/1714156966 >> v1:172.21.15.190:6789/0 conn(0x7f38a0070010 legacy=0x7f38a0135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient: _finish_hunting 0 2024-09-11T05:14:10.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 1 monclient: found mon.noname-a 2024-09-11T05:14:10.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:10.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.402+0000 7f38a4f6d640 1 -- 172.21.15.138:0/1714156966 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a009cbf0 con 0x7f38a009e6e0 2024-09-11T05:14:10.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f38a4f6d640 1 -- 172.21.15.138:0/1714156966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 msgr2=0x7f38a013d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f38a4f6d640 1 --2- 172.21.15.138:0/1714156966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 0x7f38a013d640 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f38a0135200 tx=0x7f389c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326120a3c3c81262 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 1 -- 172.21.15.138:0/1714156966 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c0a5700 con 0x7f38a009e6e0 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: dump: 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.413 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: _finish_auth 0 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: _check_auth_tickets 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 1 -- 172.21.15.138:0/1714156966 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f389c0a58a0 con 0x7f38a009e6e0 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 1 -- 172.21.15.138:0/1714156966 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c0a5c70 con 0x7f38a009e6e0 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f38a79d2640 5 monclient: authenticate success, global_id 64350 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f38a79d2640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:10.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f38a79d2640 4 set_mon_vals no callback set 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.406+0000 7f3897fff640 10 monclient: dump: 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient: shutdown 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- 172.21.15.138:0/1714156966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 msgr2=0x7f38a013d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 --2- 172.21.15.138:0/1714156966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 0x7f38a013d640 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f38a0135200 tx=0x7f389c01c880 comp rx=0 tx=0).stop 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- 172.21.15.138:0/1714156966 shutdown_connections 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 --2- 172.21.15.138:0/1714156966 >> v2:172.21.15.190:3300/0 conn(0x7f38a009eb10 0x7f38a013fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 --2- 172.21.15.138:0/1714156966 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 0x7f38a013d640 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- 172.21.15.138:0/1714156966 >> 172.21.15.138:0/1714156966 conn(0x7f38a00ab830 msgr2=0x7f38a00a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- 172.21.15.138:0/1714156966 shutdown_connections 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- 172.21.15.138:0/1714156966 wait complete. 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient: build_initial_monmap 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient: monmap: 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:10.416 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 librados: starting msgr at 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 librados: starting objecter 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 Processor -- start 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- start start 2024-09-11T05:14:10.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 librados: setting wanted keys 2024-09-11T05:14:10.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 librados: calling monclient init 2024-09-11T05:14:10.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient: init 2024-09-11T05:14:10.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:10.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:14:10.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a0070010 0x7f38a012eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient(hunting): picked mon.noname-c con 0x7f38a0070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:10.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 0x7f38a012f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient(hunting): picked mon.noname-a con 0x7f38a009e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:10.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38a009eb10 0x7f38a00a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient(hunting): picked mon.noname-b con 0x7f38a009eb10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f38a0142c80 con 0x7f38a009e6e0 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f38a0142b00 con 0x7f38a009eb10 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f38a0142e00 con 0x7f38a0070010 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:10.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a79d2640 10 monclient(hunting): authenticate will time out at 1333.849995s 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 0x7f38a012f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a0070010 0x7f38a012eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 0x7f38a012f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40438/0 (socket says 172.21.15.138:40438) 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 1 -- 172.21.15.138:0/18932457 learned_addr learned my addr 172.21.15.138:0/18932457 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient(hunting): get_auth_request con 0x7f38a0070010 auth_method 0 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 10 monclient(hunting): get_auth_request con 0x7f38a009e6e0 auth_method 0 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient(hunting): handle_auth_done global_id 22679 payload 824 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38a009eb10 msgr2=0x7f38a00a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38a009eb10 0x7f38a00a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 msgr2=0x7f38a012f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f38a009e6e0 0x7f38a012f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient: _finish_hunting 0 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 1 monclient: found mon.noname-c 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.410+0000 7f38a576e640 1 -- 172.21.15.138:0/18932457 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a00a4fc0 con 0x7f38a0070010 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a576e640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a0070010 msgr2=0x7f38a012eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a576e640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a0070010 0x7f38a012eac0 secure :-1 s=READY pgs=5782 cs=0 l=1 rev1=1 crypto rx=0x7f389000c410 tx=0x7f389000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 client.?.objecter ms_handle_connect 0x7f38a0070010 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890017070 con 0x7f38a0070010 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: dump: 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.427 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: _finish_auth 0 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38900069e0 con 0x7f38a0070010 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890006dd0 con 0x7f38a0070010 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 monclient: dump: 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.429 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38977fe640 4 set_mon_vals no callback set 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 5 monclient: authenticate success, global_id 22679 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 10 monclient: _renew_subs 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:10.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a0137870 con 0x7f38a0070010 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 10 client.22679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 10 monclient: _renew_subs 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a012f540 con 0x7f38a0070010 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 1 librados: init done 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a00a3ad0 0x7f38a00a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f38a00aacb0 con 0x7f38a00a3ad0 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a00a3ad0 0x7f38a00a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 10 monclient: get_auth_request con 0x7f38a00a3ad0 auth_method 0 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3890007030 con 0x7f38a0070010 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_done global_id 22682 payload 824 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a4f6d640 1 -- 172.21.15.138:0/18932457 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38a00a2360 con 0x7f38a00a3ad0 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 1 --2- 172.21.15.138:0/18932457 >> v2:172.21.15.190:6800/1562782189 conn(0x7f388403ced0 0x7f388403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(317..317 src has 1..317) ==== 5361+0+0 (secure 0 0 0) 0x7f389004f100 con 0x7f38a0070010 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 10 client.22679.objecter ms_dispatch 0x7f38a0000ba0 osd_map(317..317 src has 1..317) 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 3 client.22679.objecter handle_osd_map got epochs [317,317] > 0 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 3 client.22679.objecter handle_osd_map decoding full epoch 317 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f3895ffb640 20 client.22679.objecter dump_active .. 0 homeless 2024-09-11T05:14:10.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a5f6f640 1 --2- 172.21.15.138:0/18932457 >> v2:172.21.15.190:6800/1562782189 conn(0x7f388403ced0 0x7f388403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.414+0000 7f38a5f6f640 10 monclient: get_auth_request con 0x7f388403ced0 auth_method 0 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f38a4f6d640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a00a3ad0 msgr2=0x7f38a00a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 10 client.22679.objecter ms_handle_connect 0x7f38a00a3ad0 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f38a4f6d640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a00a3ad0 0x7f38a00a3eb0 secure :-1 s=READY pgs=5783 cs=0 l=1 rev1=1 crypto rx=0x7f389c01d120 tx=0x7f389c019620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 10 client.22679.objecter resend_mon_ops 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c0c0640 con 0x7f38a00a3ad0 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 10 monclient: dump: 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.433 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f389c0b63f0 con 0x7f38a00a3ad0 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a00a3ad0 msgr2=0x7f38a00a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a00a3ad0 0x7f38a00a3eb0 secure :-1 s=READY pgs=5783 cs=0 l=1 rev1=1 crypto rx=0x7f389c01d120 tx=0x7f389c019620 comp rx=0 tx=0).stop 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a00a3ad0 msgr2=0x7f38a00a3eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:10.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a00a3ad0 0x7f38a00a3eb0 secure :-1 s=CLOSED pgs=5783 cs=0 l=1 rev1=1 crypto rx=0x7f389c01d120 tx=0x7f389c019620 comp rx=0 tx=0).stop 2024-09-11T05:14:10.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f3895ffb640 10 client.22679.objecter ms_handle_connect 0x7f388403ced0 2024-09-11T05:14:10.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.418+0000 7f38a5f6f640 1 --2- 172.21.15.138:0/18932457 >> v2:172.21.15.190:6800/1562782189 conn(0x7f388403ced0 0x7f388403f390 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f389800a810 tx=0x7f3898008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a79d2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:14:10.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a79d2640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a01372b0 0x7f38a00a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a79d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f38a00a4930 con 0x7f38a01372b0 2024-09-11T05:14:10.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a01372b0 0x7f38a00a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 10 monclient: get_auth_request con 0x7f38a01372b0 auth_method 0 2024-09-11T05:14:10.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 10 monclient(hunting): handle_auth_done global_id 22685 payload 824 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.422+0000 7f38a4f6d640 1 -- 172.21.15.138:0/18932457 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f38a0069fa0 con 0x7f38a01372b0 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 10 client.22679.objecter ms_handle_connect 0x7f38a01372b0 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 10 client.22679.objecter resend_mon_ops 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a4f6d640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a01372b0 msgr2=0x7f38a00a43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a4f6d640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a01372b0 0x7f38a00a43f0 secure :-1 s=READY pgs=5784 cs=0 l=1 rev1=1 crypto rx=0x7f389c01c8c0 tx=0x7f389c0a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c0c9070 con 0x7f38a01372b0 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:14:10.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 10 monclient: dump: 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f389c0c52b0 con 0x7f38a01372b0 2024-09-11T05:14:10.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a01372b0 msgr2=0x7f38a00a43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a01372b0 0x7f38a00a43f0 secure :-1 s=READY pgs=5784 cs=0 l=1 rev1=1 crypto rx=0x7f389c01c8c0 tx=0x7f389c0a7000 comp rx=0 tx=0).stop 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a01372b0 msgr2=0x7f38a00a43f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f3895ffb640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a01372b0 0x7f38a00a43f0 secure :-1 s=CLOSED pgs=5784 cs=0 l=1 rev1=1 crypto rx=0x7f389c01c8c0 tx=0x7f389c0a7000 comp rx=0 tx=0).stop 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:10.441 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:14:10.442 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 26, 2024-09-11T05:14:10.442 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 409199, 2024-09-11T05:14:10.442 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:14:10.442 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:14:10.442 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:14:10.442 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 7, 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.443 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.444 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:14:10.445 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:10.446 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:14:10.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.449 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.450 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:10.451 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.453 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:14:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.454 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:14:10.455 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.455 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.455 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:14:10.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.455 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.456 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:14:10.457 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:14:10.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 10 librados: watch_flush enter 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 10 librados: watch_flush exit 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 20 client.22679.objecter shutdown clearing up homeless session... 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 10 client.22679.objecter successfully canceled tick 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 >> v2:172.21.15.190:6800/1562782189 conn(0x7f388403ced0 msgr2=0x7f388403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 --2- 172.21.15.138:0/18932457 >> v2:172.21.15.190:6800/1562782189 conn(0x7f388403ced0 0x7f388403f390 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f389800a810 tx=0x7f3898008040 comp rx=0 tx=0).stop 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 10 monclient: shutdown 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a0070010 msgr2=0x7f38a012eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 --2- 172.21.15.138:0/18932457 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f38a0070010 0x7f38a012eac0 secure :-1 s=READY pgs=5782 cs=0 l=1 rev1=1 crypto rx=0x7f389000c410 tx=0x7f389000c8d0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a5f6f640 1 -- 172.21.15.138:0/18932457 reap_dead start 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a5f6f640 1 -- 172.21.15.138:0/18932457 reap_dead start 2024-09-11T05:14:10.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 shutdown_connections 2024-09-11T05:14:10.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 >> 172.21.15.138:0/18932457 conn(0x7f38a00ab830 msgr2=0x7f38a006ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:10.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 shutdown_connections 2024-09-11T05:14:10.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 -- 172.21.15.138:0/18932457 wait complete. 2024-09-11T05:14:10.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.426+0000 7f38a79d2640 1 librados: shutdown 2024-09-11T05:14:10.460 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:14:10.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:14:10.460 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:14:10.586 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient: get_monmap_and_config 2024-09-11T05:14:10.586 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient: build_initial_monmap 2024-09-11T05:14:10.586 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient: monmap: 2024-09-11T05:14:10.586 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:10.586 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.586 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:10.587634+0000 2024-09-11T05:14:10.586 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:10.587634+0000 2024-09-11T05:14:10.587 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:10.587 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.587 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:10.587 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:10.587 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:10.587 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:10.587 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:10.587 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 1 Processor -- start 2024-09-11T05:14:10.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 1 -- start start 2024-09-11T05:14:10.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient: init 2024-09-11T05:14:10.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:10.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-11T05:14:10.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6d580a48e0 0x7f6d580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient(hunting): picked mon.noname-c con 0x7f6d580a48e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:10.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d580a5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient(hunting): picked mon.noname-b con 0x7f6d5809fc60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:10.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6d580a06b0 0x7f6d58136700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient(hunting): picked mon.noname-a con 0x7f6d580a06b0 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:10.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f6d580a1ca0 con 0x7f6d580a06b0 2024-09-11T05:14:10.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6d580a5740 con 0x7f6d580a48e0 2024-09-11T05:14:10.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6d580a0b70 con 0x7f6d5809fc60 2024-09-11T05:14:10.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:10.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d6035e640 10 monclient(hunting): authenticate will time out at 1334.023582s 2024-09-11T05:14:10.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d580a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6d580a06b0 0x7f6d58136700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6d580a06b0 0x7f6d58136700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40444/0 (socket says 172.21.15.138:40444) 2024-09-11T05:14:10.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d580a5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:54604/0 (socket says 172.21.15.138:54604) 2024-09-11T05:14:10.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/1836650177 learned_addr learned my addr 172.21.15.138:0/1836650177 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:10.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 10 monclient(hunting): get_auth_request con 0x7f6d580a06b0 auth_method 0 2024-09-11T05:14:10.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 10 monclient(hunting): get_auth_request con 0x7f6d5809fc60 auth_method 0 2024-09-11T05:14:10.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e8fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5d8f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.582+0000 7f6d5e0fa640 1 --2- 172.21.15.138:0/1836650177 >> v2:172.21.15.190:3300/0 conn(0x7f6d580a48e0 0x7f6d580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e0fa640 10 monclient(hunting): get_auth_request con 0x7f6d580a48e0 auth_method 0 2024-09-11T05:14:10.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e0fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 10 monclient(hunting): handle_auth_done global_id 64356 payload 274 2024-09-11T05:14:10.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 msgr2=0x7f6d580a5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:10.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 1 --2- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d580a5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/1836650177 >> v2:172.21.15.190:3300/0 conn(0x7f6d580a48e0 msgr2=0x7f6d580a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:10.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 1 --2- 172.21.15.138:0/1836650177 >> v2:172.21.15.190:3300/0 conn(0x7f6d580a48e0 0x7f6d580a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 10 monclient: _finish_hunting 0 2024-09-11T05:14:10.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 1 monclient: found mon.noname-a 2024-09-11T05:14:10.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:10.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/1836650177 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d580a21d0 con 0x7f6d580a06b0 2024-09-11T05:14:10.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.586+0000 7f6d5d8f9640 1 --2- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d580a5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:14:10.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a06b0 msgr2=0x7f6d58136700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 --2- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a06b0 0x7f6d58136700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6d580665b0 tx=0x7f6d5001c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802177a229e6c9cd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 1 -- 172.21.15.138:0/1836650177 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500ac020 con 0x7f6d580a06b0 2024-09-11T05:14:10.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: dump: 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: _finish_auth 0 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: _check_auth_tickets 2024-09-11T05:14:10.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 1 -- 172.21.15.138:0/1836650177 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d500a6470 con 0x7f6d580a06b0 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 1 -- 172.21.15.138:0/1836650177 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500a6840 con 0x7f6d580a06b0 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 5 monclient: authenticate success, global_id 64356 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 4 set_mon_vals no callback set 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d0f8640 10 monclient: dump: 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.607 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient: shutdown 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a06b0 msgr2=0x7f6d58136700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 --2- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a06b0 0x7f6d58136700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6d580665b0 tx=0x7f6d5001c5c0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- 172.21.15.138:0/1836650177 shutdown_connections 2024-09-11T05:14:10.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 --2- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a06b0 0x7f6d58136700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 --2- 172.21.15.138:0/1836650177 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d580a5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 --2- 172.21.15.138:0/1836650177 >> v2:172.21.15.190:3300/0 conn(0x7f6d580a48e0 0x7f6d580a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- 172.21.15.138:0/1836650177 >> 172.21.15.138:0/1836650177 conn(0x7f6d580a9590 msgr2=0x7f6d580a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- 172.21.15.138:0/1836650177 shutdown_connections 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- 172.21.15.138:0/1836650177 wait complete. 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient: build_initial_monmap 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient: monmap: 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:10.609 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 librados: starting msgr at 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 librados: starting objecter 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 Processor -- start 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- start start 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 librados: setting wanted keys 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 librados: calling monclient init 2024-09-11T05:14:10.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient: init 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d581d0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient(hunting): picked mon.noname-c con 0x7f6d5809fc60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d580a06b0 0x7f6d58135c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient(hunting): picked mon.noname-b con 0x7f6d580a06b0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a48e0 0x7f6d58136140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient(hunting): picked mon.noname-a con 0x7f6d580a48e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5813fdc0 con 0x7f6d580a48e0 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5813fc40 con 0x7f6d580a06b0 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6d5813ff40 con 0x7f6d5809fc60 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d6035e640 10 monclient(hunting): authenticate will time out at 1334.030781s 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d581d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d581d0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:54606/0 (socket says 172.21.15.138:54606) 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 1 -- 172.21.15.138:0/468189821 learned_addr learned my addr 172.21.15.138:0/468189821 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 10 monclient(hunting): get_auth_request con 0x7f6d5809fc60 auth_method 0 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a48e0 0x7f6d58136140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient(hunting): get_auth_request con 0x7f6d580a48e0 auth_method 0 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d8f9640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d580a06b0 0x7f6d58135c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d8f9640 10 monclient(hunting): get_auth_request con 0x7f6d580a06b0 auth_method 0 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d8f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d8f9640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5d8f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient(hunting): handle_auth_done global_id 64359 payload 824 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 msgr2=0x7f6d581d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6d5809fc60 0x7f6d581d0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d580a06b0 msgr2=0x7f6d58135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d580a06b0 0x7f6d58135c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient: _finish_hunting 0 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 monclient: found mon.noname-a 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:10.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.590+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/468189821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5806fa30 con 0x7f6d580a48e0 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a48e0 msgr2=0x7f6d58136140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e8fb640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a48e0 0x7f6d58136140 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6d500aa000 tx=0x7f6d50019360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 client.?.objecter ms_handle_connect 0x7f6d580a48e0 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500ac040 con 0x7f6d580a48e0 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: dump: 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.615 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: _finish_auth 0 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: _check_auth_tickets 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 5 monclient: authenticate success, global_id 64359 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d500a7450 con 0x7f6d580a48e0 2024-09-11T05:14:10.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500a7840 con 0x7f6d580a48e0 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 10 monclient: dump: 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.617 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5c8f7640 4 set_mon_vals no callback set 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 10 monclient: _renew_subs 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d580a00c0 con 0x7f6d580a48e0 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 10 client.64359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 10 monclient: _renew_subs 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d58070ac0 con 0x7f6d580a48e0 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 1 librados: init done 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813aa60 0x7f6d5813ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6d58070740 con 0x7f6d5813aa60 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e8fb640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a48e0 0x7f6d58136140 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6d500aa000 tx=0x7f6d50019360 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f6d500ea2b0 osd_map(318..318 src has 1..318) 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f6d500a7a20 con 0x7f6d580a48e0 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813aa60 0x7f6d5813ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 10 monclient: get_auth_request con 0x7f6d5813aa60 auth_method 0 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d4effd640 1 --2- 172.21.15.138:0/468189821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d3c03ced0 0x7f6d3c03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_done global_id 74119 payload 824 2024-09-11T05:14:10.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.594+0000 7f6d5e0fa640 1 -- 172.21.15.138:0/468189821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d581d0b40 con 0x7f6d5813aa60 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d5d8f9640 1 --2- 172.21.15.138:0/468189821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d3c03ced0 0x7f6d3c03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d5d8f9640 10 monclient: get_auth_request con 0x7f6d3c03ced0 auth_method 0 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d5e0fa640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813aa60 msgr2=0x7f6d5813ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 10 client.64359.objecter ms_handle_connect 0x7f6d5813aa60 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 10 client.64359.objecter resend_mon_ops 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d5e0fa640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813aa60 0x7f6d5813ae40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d54010820 tx=0x7f6d54010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54018070 con 0x7f6d5813aa60 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 10 monclient: dump: 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.620 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 10 client.64359.objecter ms_handle_connect 0x7f6d3c03ced0 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d5d8f9640 1 --2- 172.21.15.138:0/468189821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d3c03ced0 0x7f6d3c03f390 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009580 tx=0x7f6d48008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d5400ebf0 con 0x7f6d5813aa60 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813aa60 msgr2=0x7f6d5813ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813aa60 0x7f6d5813ae40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d54010820 tx=0x7f6d54010ce0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813aa60 msgr2=0x7f6d5813ae40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:10.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.598+0000 7f6d4effd640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813aa60 0x7f6d5813ae40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d6035e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:14:10.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d6035e640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813b380 0x7f6d580a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d6035e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6d58136700 con 0x7f6d5813b380 2024-09-11T05:14:10.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813b380 0x7f6d580a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 10 monclient: get_auth_request con 0x7f6d5813b380 auth_method 0 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 10 monclient(hunting): handle_auth_done global_id 74122 payload 824 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.606+0000 7f6d5e0fa640 1 -- 172.21.15.138:0/468189821 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6d5806cbf0 con 0x7f6d5813b380 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d5e0fa640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813b380 msgr2=0x7f6d580a6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d5e0fa640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813b380 0x7f6d580a6dd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d5401a000 tx=0x7f6d540046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 client.64359.objecter ms_handle_connect 0x7f6d5813b380 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 client.64359.objecter resend_mon_ops 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5401d450 con 0x7f6d5813b380 2024-09-11T05:14:10.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 monclient: dump: 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f6d5401d450 con 0x7f6d5813b380 2024-09-11T05:14:10.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813b380 msgr2=0x7f6d580a6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813b380 0x7f6d580a6dd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d5401a000 tx=0x7f6d540046d0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813b380 msgr2=0x7f6d580a6dd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d5813b380 0x7f6d580a6dd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 26, 2024-09-11T05:14:10.626 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 18163, 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 7, 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.627 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:14:10.628 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:14:10.630 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:10.631 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.632 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.633 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:14:10.634 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:10.636 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.637 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:10.638 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:10.639 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:10.639 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:10.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:10.639 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:14:10.639 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:14:10.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 10 librados: watch_flush enter 2024-09-11T05:14:10.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 10 librados: watch_flush exit 2024-09-11T05:14:10.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 20 client.64359.objecter shutdown clearing up homeless session... 2024-09-11T05:14:10.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 10 client.64359.objecter successfully canceled tick 2024-09-11T05:14:10.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d3c03ced0 msgr2=0x7f6d3c03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 --2- 172.21.15.138:0/468189821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6d3c03ced0 0x7f6d3c03f390 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009580 tx=0x7f6d48008040 comp rx=0 tx=0).stop 2024-09-11T05:14:10.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 10 monclient: shutdown 2024-09-11T05:14:10.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a48e0 msgr2=0x7f6d58136140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 --2- 172.21.15.138:0/468189821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6d580a48e0 0x7f6d58136140 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6d500aa000 tx=0x7f6d50019360 comp rx=0 tx=0).stop 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/468189821 reap_dead start 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d5e8fb640 1 -- 172.21.15.138:0/468189821 reap_dead start 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 1 -- 172.21.15.138:0/468189821 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(318..318 src has 1..318) ==== 5374+0+0 (unknown 0 0 0) 0x7f6d500ea2b0 con 0x7f6d580a48e0 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 client.64359.objecter ms_dispatch 0x7f6d58000ba0 osd_map(318..318 src has 1..318) 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d4effd640 10 librados: disconnected, discarding osd_map(318..318 src has 1..318) 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 shutdown_connections 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 >> 172.21.15.138:0/468189821 conn(0x7f6d580a9590 msgr2=0x7f6d5809e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 shutdown_connections 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 -- 172.21.15.138:0/468189821 wait complete. 2024-09-11T05:14:10.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.610+0000 7f6d6035e640 1 librados: shutdown 2024-09-11T05:14:10.643 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:14:10.643 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-11T05:14:10.643 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T05:14:10.643 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-11T05:14:10.643 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-11T05:14:10.643 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T05:14:10.644 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:14:10.658 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T05:14:10.659 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:14:10.659 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient: get_monmap_and_config 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient: build_initial_monmap 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient: monmap: 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:10.818509+0000 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:10.818509+0000 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:10.817 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:10.818 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:10.818 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:10.818 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:10.818 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 1 Processor -- start 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 1 -- start start 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient: init 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe6400a5b40 0x7fe6400a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient(hunting): picked mon.noname-c con 0x7fe6400a5b40 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe6400a7130 0x7fe6400a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient(hunting): picked mon.noname-a con 0x7fe6400a7130 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe64009d840 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64009d490 con 0x7fe64009d840 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fe64009e110 con 0x7fe6400a7130 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe6400a69a0 con 0x7fe6400a5b40 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe647592640 10 monclient(hunting): authenticate will time out at 1334.255073s 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe6400a7130 0x7fe6400a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe6400a7130 0x7fe6400a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40472/0 (socket says 172.21.15.138:40472) 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 1 -- 172.21.15.138:0/3514783330 learned_addr learned my addr 172.21.15.138:0/3514783330 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:10.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient(hunting): get_auth_request con 0x7fe6400a7130 auth_method 0 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe64532e640 1 --2- 172.21.15.138:0/3514783330 >> v2:172.21.15.190:3300/0 conn(0x7fe6400a5b40 0x7fe6400a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe64532e640 10 monclient(hunting): get_auth_request con 0x7fe6400a5b40 auth_method 0 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe64532e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe64532e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe64532e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient(hunting): handle_auth_done global_id 64362 payload 274 2024-09-11T05:14:10.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 1 -- 172.21.15.138:0/3514783330 >> v2:172.21.15.190:3300/0 conn(0x7fe6400a5b40 msgr2=0x7fe6400a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:10.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 1 --2- 172.21.15.138:0/3514783330 >> v2:172.21.15.190:3300/0 conn(0x7fe6400a5b40 0x7fe6400a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 1 -- 172.21.15.138:0/3514783330 >> v1:172.21.15.190:6789/0 conn(0x7fe64009d840 legacy=0x7fe640138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:10.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient: _finish_hunting 0 2024-09-11T05:14:10.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 1 monclient: found mon.noname-a 2024-09-11T05:14:10.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:10.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.814+0000 7fe644b2d640 1 -- 172.21.15.138:0/3514783330 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe640070580 con 0x7fe6400a7130 2024-09-11T05:14:10.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe644b2d640 1 -- 172.21.15.138:0/3514783330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6400a7130 msgr2=0x7fe6400a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe644b2d640 1 --2- 172.21.15.138:0/3514783330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6400a7130 0x7fe6400a6460 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe6400524c0 tx=0x7fe63c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d748a92069913a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 1 -- 172.21.15.138:0/3514783330 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c0b0070 con 0x7fe6400a7130 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: dump: 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.832 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: _finish_auth 0 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: _check_auth_tickets 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 1 -- 172.21.15.138:0/3514783330 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63c0ab600 con 0x7fe6400a7130 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 5 monclient: authenticate success, global_id 64362 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 4 set_mon_vals no callback set 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 1 -- 172.21.15.138:0/3514783330 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c0a4e60 con 0x7fe6400a7130 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe637fff640 10 monclient: dump: 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient: shutdown 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:10.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- 172.21.15.138:0/3514783330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6400a7130 msgr2=0x7fe6400a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 --2- 172.21.15.138:0/3514783330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6400a7130 0x7fe6400a6460 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe6400524c0 tx=0x7fe63c01c3e0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- 172.21.15.138:0/3514783330 shutdown_connections 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 --2- 172.21.15.138:0/3514783330 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe6400a7130 0x7fe6400a6460 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 --2- 172.21.15.138:0/3514783330 >> v2:172.21.15.190:3300/0 conn(0x7fe6400a5b40 0x7fe6400a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- 172.21.15.138:0/3514783330 >> 172.21.15.138:0/3514783330 conn(0x7fe6400aba10 msgr2=0x7fe6400a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- 172.21.15.138:0/3514783330 shutdown_connections 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- 172.21.15.138:0/3514783330 wait complete. 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient: build_initial_monmap 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient: monmap: 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:10.835 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 librados: starting msgr at 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 librados: starting objecter 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 Processor -- start 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- start start 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 librados: setting wanted keys 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 librados: calling monclient init 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient: init 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:10.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:14:10.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe64009d840 0x7fe6401d0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient(hunting): picked mon.noname-a con 0x7fe64009d840 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:10.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6400a5b40 0x7fe640137a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient(hunting): picked mon.noname-b con 0x7fe6400a5b40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:10.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe6400a7130 0x7fe64013da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient(hunting): picked mon.noname-c con 0x7fe6400a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe640142f60 con 0x7fe64009d840 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe640142de0 con 0x7fe6400a5b40 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe6401430e0 con 0x7fe6400a7130 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe647592640 10 monclient(hunting): authenticate will time out at 1334.262567s 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe645b2f640 1 -- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe6400a7130 msgr2=0x7fe64013da10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe645b2f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe6400a7130 0x7fe64013da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:10.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe64009d840 0x7fe6401d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe64009d840 0x7fe6401d0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40480/0 (socket says 172.21.15.138:40480) 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 -- 172.21.15.138:0/3050922810 learned_addr learned my addr 172.21.15.138:0/3050922810 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient(hunting): get_auth_request con 0x7fe64009d840 auth_method 0 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient(hunting): handle_auth_done global_id 64365 payload 824 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe644b2d640 1 --2- 172.21.15.138:0/3050922810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6400a5b40 0x7fe640137a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 -- 172.21.15.138:0/3050922810 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe6400a7130 msgr2=0x7fe64013da10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 --2- 172.21.15.138:0/3050922810 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe6400a7130 0x7fe64013da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 -- 172.21.15.138:0/3050922810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6400a5b40 msgr2=0x7fe640137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 --2- 172.21.15.138:0/3050922810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6400a5b40 0x7fe640137a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient: _finish_hunting 0 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 monclient: found mon.noname-a 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.822+0000 7fe64532e640 1 -- 172.21.15.138:0/3050922810 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe640121890 con 0x7fe64009d840 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe64532e640 1 -- 172.21.15.138:0/3050922810 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe64009d840 msgr2=0x7fe6401d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe64532e640 1 --2- 172.21.15.138:0/3050922810 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe64009d840 0x7fe6401d0460 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe63000d800 tx=0x7fe63000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 client.?.objecter ms_handle_connect 0x7fe64009d840 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 1 -- 172.21.15.138:0/3050922810 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63000b8d0 con 0x7fe64009d840 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:10.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: dump: 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: _finish_auth 0 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:14:10.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 5 monclient: authenticate success, global_id 64365 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 1 -- 172.21.15.138:0/3050922810 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe630019070 con 0x7fe64009d840 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 1 -- 172.21.15.138:0/3050922810 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63000bc70 con 0x7fe64009d840 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 monclient: dump: 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:10.842 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe636ffd640 4 set_mon_vals no callback set 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 10 monclient: _renew_subs 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe640121b70 con 0x7fe64009d840 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 10 client.64365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 10 monclient: _renew_subs 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64013ffb0 con 0x7fe64009d840 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 1 librados: init done 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 1 -- 172.21.15.138:0/3050922810 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fe630013070 con 0x7fe64009d840 2024-09-11T05:14:10.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe640122090 con 0x7fe64009d840 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 1 --2- 172.21.15.138:0/3050922810 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe62403ce50 0x7fe62403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 1 -- 172.21.15.138:0/3050922810 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(318..318 src has 1..318) ==== 5374+0+0 (secure 0 0 0) 0x7fe63004fbb0 con 0x7fe64009d840 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 10 client.64365.objecter ms_dispatch 0x7fe640000ba0 osd_map(318..318 src has 1..318) 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 3 client.64365.objecter handle_osd_map got epochs [318,318] > 0 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 3 client.64365.objecter handle_osd_map decoding full epoch 318 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe635ffb640 20 client.64365.objecter dump_active .. 0 homeless 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe644b2d640 1 --2- 172.21.15.138:0/3050922810 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe62403ce50 0x7fe62403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:10.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.826+0000 7fe644b2d640 10 monclient: get_auth_request con 0x7fe62403ce50 auth_method 0 2024-09-11T05:14:10.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.830+0000 7fe644b2d640 1 --2- 172.21.15.138:0/3050922810 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe62403ce50 0x7fe62403f310 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7fe64006cd40 tx=0x7fe63c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:10.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.830+0000 7fe635ffb640 10 client.64365.objecter ms_handle_connect 0x7fe62403ce50 2024-09-11T05:14:10.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.830+0000 7fe635ffb640 1 -- 172.21.15.138:0/3050922810 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe63000be10 con 0x7fe64009d840 2024-09-11T05:14:10.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.830+0000 7fe635ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:10.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:10.830+0000 7fe635ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:11.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.066+0000 7fe647592640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:14:11.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.066+0000 7fe647592640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:11.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.066+0000 7fe647592640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:11.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.066+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe640138900 con 0x7fe64009d840 2024-09-11T05:14:11.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.066+0000 7fe635ffb640 1 -- 172.21.15.138:0/3050922810 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fe63004e360 con 0x7fe64009d840 2024-09-11T05:14:11.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.066+0000 7fe635ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:11.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.066+0000 7fe635ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:11.071 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:14:11.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 10 librados: watch_flush enter 2024-09-11T05:14:11.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 10 librados: watch_flush exit 2024-09-11T05:14:11.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 20 client.64365.objecter shutdown clearing up homeless session... 2024-09-11T05:14:11.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 10 client.64365.objecter successfully canceled tick 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe62403ce50 msgr2=0x7fe62403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 --2- 172.21.15.138:0/3050922810 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe62403ce50 0x7fe62403f310 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7fe64006cd40 tx=0x7fe63c008040 comp rx=0 tx=0).stop 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 10 monclient: shutdown 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe64009d840 msgr2=0x7fe6401d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 --2- 172.21.15.138:0/3050922810 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe64009d840 0x7fe6401d0460 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe63000d800 tx=0x7fe63000dcc0 comp rx=0 tx=0).stop 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 shutdown_connections 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 --2- 172.21.15.138:0/3050922810 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe62403ce50 0x7fe62403f310 unknown :-1 s=CLOSED pgs=3407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 --2- 172.21.15.138:0/3050922810 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe6400a7130 0x7fe64013da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 --2- 172.21.15.138:0/3050922810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6400a5b40 0x7fe640137a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 --2- 172.21.15.138:0/3050922810 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe64009d840 0x7fe6401d0460 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 >> 172.21.15.138:0/3050922810 conn(0x7fe6400aba10 msgr2=0x7fe6400a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 shutdown_connections 2024-09-11T05:14:11.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 -- 172.21.15.138:0/3050922810 wait complete. 2024-09-11T05:14:11.075 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:11.070+0000 7fe647592640 1 librados: shutdown 2024-09-11T05:14:14.091 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:14:14.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient: get_monmap_and_config 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient: build_initial_monmap 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient: monmap: 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:14.219421+0000 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:14.219421+0000 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:14.218 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:14.219 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:14.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 1 Processor -- start 2024-09-11T05:14:14.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 1 -- start start 2024-09-11T05:14:14.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient: init 2024-09-11T05:14:14.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:14.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-11T05:14:14.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f0406e0e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:14:14.221 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f04136e30 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f7f04136170 0x7f7f0413ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient(hunting): picked mon.noname-a con 0x7f7f04136170 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f040a3270 con 0x7f7f0406e0e0 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f040a0af0 con 0x7f7f04136e30 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f7f0406e4c0 con 0x7f7f04136170 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0c56f640 10 monclient(hunting): authenticate will time out at 1337.655491s 2024-09-11T05:14:14.222 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0a30b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f7f0406e0e0 0x7f7f04136720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:41460/0 (socket says 172.21.15.138:41460) 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0a30b640 1 -- 172.21.15.138:0/3980166115 learned_addr learned my addr 172.21.15.138:0/3980166115 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0ab0c640 1 --2- 172.21.15.138:0/3980166115 >> v2:172.21.15.138:3300/0 conn(0x7f7f04136170 0x7f7f0413ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0ab0c640 10 monclient(hunting): get_auth_request con 0x7f7f04136170 auth_method 0 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0ab0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0ab0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0ab0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0ab0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0ab0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.214+0000 7f7f0ab0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.218+0000 7f7f0ab0c640 10 monclient(hunting): handle_auth_done global_id 64368 payload 274 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.218+0000 7f7f0ab0c640 1 -- 172.21.15.138:0/3980166115 >> v1:172.21.15.190:6789/0 conn(0x7f7f04136e30 legacy=0x7f7f0413b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.218+0000 7f7f0ab0c640 1 -- 172.21.15.138:0/3980166115 >> v1:172.21.15.138:6789/0 conn(0x7f7f0406e0e0 legacy=0x7f7f04136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.218+0000 7f7f0ab0c640 10 monclient: _finish_hunting 0 2024-09-11T05:14:14.223 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.218+0000 7f7f0ab0c640 1 monclient: found mon.noname-a 2024-09-11T05:14:14.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.218+0000 7f7f0ab0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:14.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.218+0000 7f7f0ab0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:14.224 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.218+0000 7f7f0ab0c640 1 -- 172.21.15.138:0/3980166115 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f040a2f90 con 0x7f7f04136170 2024-09-11T05:14:14.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0ab0c640 1 -- 172.21.15.138:0/3980166115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f04136170 msgr2=0x7f7f0413ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:14.227 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0ab0c640 1 --2- 172.21.15.138:0/3980166115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f04136170 0x7f7f0413ec00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7efc009d90 tx=0x7f7efc01c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb28e2ab45fe934 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 1 -- 172.21.15.138:0/3980166115 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc0a86f0 con 0x7f7f04136170 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: dump: 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:14.228 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: _finish_auth 0 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: _check_auth_tickets 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 1 -- 172.21.15.138:0/3980166115 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7efc0a8890 con 0x7f7f04136170 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 1 -- 172.21.15.138:0/3980166115 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc0a8c60 con 0x7f7f04136170 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09309640 10 monclient: dump: 2024-09-11T05:14:14.229 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 5 monclient: authenticate success, global_id 64368 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 4 set_mon_vals no callback set 2024-09-11T05:14:14.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient: shutdown 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- 172.21.15.138:0/3980166115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f04136170 msgr2=0x7f7f0413ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 --2- 172.21.15.138:0/3980166115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f04136170 0x7f7f0413ec00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7efc009d90 tx=0x7f7efc01c650 comp rx=0 tx=0).stop 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- 172.21.15.138:0/3980166115 shutdown_connections 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 --2- 172.21.15.138:0/3980166115 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f04136170 0x7f7f0413ec00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- 172.21.15.138:0/3980166115 >> 172.21.15.138:0/3980166115 conn(0x7f7f04063640 msgr2=0x7f7f04063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- 172.21.15.138:0/3980166115 shutdown_connections 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- 172.21.15.138:0/3980166115 wait complete. 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient: build_initial_monmap 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient: monmap: 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:14.231 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 librados: starting msgr at 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 librados: starting objecter 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 Processor -- start 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- start start 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 librados: setting wanted keys 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 librados: calling monclient init 2024-09-11T05:14:14.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient: init 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 0x7f7f041ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient(hunting): picked mon.noname-a con 0x7f7f0406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f04136170 0x7f7f0413e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient(hunting): picked mon.noname-b con 0x7f7f04136170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f04136e30 0x7f7f0413e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient(hunting): picked mon.noname-c con 0x7f7f04136e30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7f04141cf0 con 0x7f7f0406e0e0 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7f04141b70 con 0x7f7f04136170 2024-09-11T05:14:14.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7f04141e70 con 0x7f7f04136e30 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0c56f640 10 monclient(hunting): authenticate will time out at 1337.662897s 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0ab0c640 1 -- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f04136e30 msgr2=0x7f7f0413e950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0ab0c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f04136e30 0x7f7f0413e950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 0x7f7f041ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 0x7f7f041ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40488/0 (socket says 172.21.15.138:40488) 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 -- 172.21.15.138:0/2964061821 learned_addr learned my addr 172.21.15.138:0/2964061821 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient(hunting): get_auth_request con 0x7f7f0406e0e0 auth_method 0 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:14.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient(hunting): handle_auth_done global_id 64371 payload 824 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 -- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f04136e30 msgr2=0x7f7f0413e950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f04136e30 0x7f7f0413e950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 -- 172.21.15.138:0/2964061821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f04136170 msgr2=0x7f7f0413e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f09b0a640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f04136170 0x7f7f0413e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f04136170 0x7f7f0413e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient: _finish_hunting 0 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 monclient: found mon.noname-a 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:14.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.222+0000 7f7f0a30b640 1 -- 172.21.15.138:0/2964061821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f04135920 con 0x7f7f0406e0e0 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0a30b640 1 -- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 msgr2=0x7f7f041ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0a30b640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 0x7f7f041ce4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400b430 tx=0x7f7ef400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 client.?.objecter ms_handle_connect 0x7f7f0406e0e0 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0a30b640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 0x7f7f041ce4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400b430 tx=0x7f7ef400b8f0 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f7ef4015020 mon_map magic: 0 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 1 -- 172.21.15.138:0/2964061821 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4015020 con 0x7f7f0406e0e0 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: dump: 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:14.238 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: _finish_auth 0 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: _check_auth_tickets 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 5 monclient: authenticate success, global_id 64371 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 1 -- 172.21.15.138:0/2964061821 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef4011a60 con 0x7f7f0406e0e0 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 10 monclient: _renew_subs 2024-09-11T05:14:14.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0409e110 con 0x7f7f0406e0e0 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efbfff640 4 set_mon_vals no callback set 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 10 client.64371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 10 monclient: _renew_subs 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0413a620 con 0x7f7f0406e0e0 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 1 librados: init done 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f04136720 con 0x7f7f0406e0e0 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 1 -- 172.21.15.138:0/2964061821 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4011e50 con 0x7f7f0406e0e0 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:14.240 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 10 monclient: dump: 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 1 -- 172.21.15.138:0/2964061821 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f7ef401a400 con 0x7f7f0406e0e0 2024-09-11T05:14:14.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7efaffd640 1 --2- 172.21.15.138:0/2964061821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7ee803cff0 0x7f7ee803f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f09b0a640 1 --2- 172.21.15.138:0/2964061821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7ee803cff0 0x7f7ee803f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.226+0000 7f7f09b0a640 10 monclient: get_auth_request con 0x7f7ee803cff0 auth_method 0 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.230+0000 7f7efaffd640 10 client.64371.objecter ms_handle_connect 0x7f7ee803cff0 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.230+0000 7f7f09b0a640 1 --2- 172.21.15.138:0/2964061821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7ee803cff0 0x7f7ee803f4b0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f7f00007fd0 tx=0x7f7f00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.230+0000 7f7efaffd640 1 -- 172.21.15.138:0/2964061821 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(318..318 src has 1..318) ==== 5374+0+0 (secure 0 0 0) 0x7f7ef404fdc0 con 0x7f7f0406e0e0 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.230+0000 7f7efaffd640 10 client.64371.objecter ms_dispatch 0x7f7f04000ba0 osd_map(318..318 src has 1..318) 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.230+0000 7f7efaffd640 3 client.64371.objecter handle_osd_map got epochs [318,318] > 0 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.230+0000 7f7efaffd640 3 client.64371.objecter handle_osd_map decoding full epoch 318 2024-09-11T05:14:14.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.230+0000 7f7efaffd640 20 client.64371.objecter dump_active .. 0 homeless 2024-09-11T05:14:14.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.854+0000 7f7efaffd640 1 -- 172.21.15.138:0/2964061821 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f7ef401f020 con 0x7f7f0406e0e0 2024-09-11T05:14:14.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.854+0000 7f7efaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:14.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:14.854+0000 7f7efaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:15.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:14:15.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:15.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:15.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7f0413af80 con 0x7f7f0406e0e0 2024-09-11T05:14:15.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7efaffd640 1 -- 172.21.15.138:0/2964061821 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f7ef404e570 con 0x7f7f0406e0e0 2024-09-11T05:14:15.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7efaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:15.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7efaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:15.100 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:14:15.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 10 librados: watch_flush enter 2024-09-11T05:14:15.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 10 librados: watch_flush exit 2024-09-11T05:14:15.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 20 client.64371.objecter shutdown clearing up homeless session... 2024-09-11T05:14:15.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 10 client.64371.objecter successfully canceled tick 2024-09-11T05:14:15.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7ee803cff0 msgr2=0x7f7ee803f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:15.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 --2- 172.21.15.138:0/2964061821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7ee803cff0 0x7f7ee803f4b0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f7f00007fd0 tx=0x7f7f00009040 comp rx=0 tx=0).stop 2024-09-11T05:14:15.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 10 monclient: shutdown 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 msgr2=0x7f7f041ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 0x7f7f041ce4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400b430 tx=0x7f7ef400b8f0 comp rx=0 tx=0).stop 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 shutdown_connections 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 --2- 172.21.15.138:0/2964061821 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7ee803cff0 0x7f7ee803f4b0 unknown :-1 s=CLOSED pgs=3408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f04136e30 0x7f7f0413e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f04136170 0x7f7f0413e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 --2- 172.21.15.138:0/2964061821 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f0406e0e0 0x7f7f041ce4c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 >> 172.21.15.138:0/2964061821 conn(0x7f7f04063640 msgr2=0x7f7f04132f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 shutdown_connections 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 -- 172.21.15.138:0/2964061821 wait complete. 2024-09-11T05:14:15.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:15.098+0000 7f7f0c56f640 1 librados: shutdown 2024-09-11T05:14:18.121 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:14:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient: get_monmap_and_config 2024-09-11T05:14:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient: build_initial_monmap 2024-09-11T05:14:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient: monmap: 2024-09-11T05:14:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:18.250621+0000 2024-09-11T05:14:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:18.250621+0000 2024-09-11T05:14:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:18.250 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:18.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 1 Processor -- start 2024-09-11T05:14:18.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 1 -- start start 2024-09-11T05:14:18.252 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient: init 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b480a0a90 0x7f5b480a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient(hunting): picked mon.noname-b con 0x7f5b480a0a90 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5b4809dc20 0x7f5b480a2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient(hunting): picked mon.noname-a con 0x7f5b4809dc20 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5b480a2bb0 0x7f5b481348c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient(hunting): picked mon.noname-c con 0x7f5b480a2bb0 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f5b480a37b0 con 0x7f5b4809dc20 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:18.253 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5b4809efa0 con 0x7f5b480a2bb0 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5b4809ecc0 con 0x7f5b480a0a90 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b501c1640 10 monclient(hunting): authenticate will time out at 1341.686887s 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4df5d640 1 -- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b480a0a90 msgr2=0x7f5b480a0e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4df5d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b480a0a90 0x7f5b480a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5b4809dc20 0x7f5b480a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5b4809dc20 0x7f5b480a2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:39686/0 (socket says 172.21.15.138:39686) 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 -- 172.21.15.138:0/1133436818 learned_addr learned my addr 172.21.15.138:0/1133436818 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient(hunting): get_auth_request con 0x7f5b4809dc20 auth_method 0 2024-09-11T05:14:18.254 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient(hunting): handle_auth_done global_id 64374 payload 274 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4e75e640 1 --2- 172.21.15.138:0/1133436818 >> v2:172.21.15.190:3300/0 conn(0x7f5b480a2bb0 0x7f5b481348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 -- 172.21.15.138:0/1133436818 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b480a0a90 msgr2=0x7f5b480a0e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 --2- 172.21.15.138:0/1133436818 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b480a0a90 0x7f5b480a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 -- 172.21.15.138:0/1133436818 >> v2:172.21.15.190:3300/0 conn(0x7f5b480a2bb0 msgr2=0x7f5b481348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 --2- 172.21.15.138:0/1133436818 >> v2:172.21.15.190:3300/0 conn(0x7f5b480a2bb0 0x7f5b481348c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient: _finish_hunting 0 2024-09-11T05:14:18.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 monclient: found mon.noname-a 2024-09-11T05:14:18.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:18.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:18.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.246+0000 7f5b4d75c640 1 -- 172.21.15.138:0/1133436818 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b480a13b0 con 0x7f5b4809dc20 2024-09-11T05:14:18.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4d75c640 1 -- 172.21.15.138:0/1133436818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 msgr2=0x7f5b480a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:18.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4d75c640 1 --2- 172.21.15.138:0/1133436818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b480a2670 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806a140 tx=0x7f5b3c01d5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8868a022783248f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:18.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 1 -- 172.21.15.138:0/1133436818 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0ab750 con 0x7f5b4809dc20 2024-09-11T05:14:18.262 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: dump: 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:18.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: _finish_auth 0 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: _check_auth_tickets 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b501c1640 5 monclient: authenticate success, global_id 64374 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b501c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 1 -- 172.21.15.138:0/1133436818 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b3c0b0070 con 0x7f5b4809dc20 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 1 -- 172.21.15.138:0/1133436818 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0abb10 con 0x7f5b4809dc20 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b4cf5b640 10 monclient: dump: 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:18.264 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b501c1640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.250+0000 7f5b501c1640 4 set_mon_vals no callback set 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient: shutdown 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:18.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- 172.21.15.138:0/1133436818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 msgr2=0x7f5b480a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 --2- 172.21.15.138:0/1133436818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b480a2670 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806a140 tx=0x7f5b3c01d5a0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- 172.21.15.138:0/1133436818 shutdown_connections 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 --2- 172.21.15.138:0/1133436818 >> v2:172.21.15.190:3300/0 conn(0x7f5b480a2bb0 0x7f5b481348c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 --2- 172.21.15.138:0/1133436818 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b480a2670 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 --2- 172.21.15.138:0/1133436818 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b480a0a90 0x7f5b480a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- 172.21.15.138:0/1133436818 >> 172.21.15.138:0/1133436818 conn(0x7f5b48063640 msgr2=0x7f5b48063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- 172.21.15.138:0/1133436818 shutdown_connections 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- 172.21.15.138:0/1133436818 wait complete. 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient: build_initial_monmap 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient: monmap: 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:18.266 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 librados: starting msgr at 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 librados: starting objecter 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 Processor -- start 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- start start 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 librados: setting wanted keys 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 librados: calling monclient init 2024-09-11T05:14:18.267 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient: init 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b481d53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient(hunting): picked mon.noname-a con 0x7f5b4809dc20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b480a0a90 0x7f5b481d5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient(hunting): picked mon.noname-b con 0x7f5b480a0a90 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b4813a3a0 0x7f5b4813a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient(hunting): picked mon.noname-c con 0x7f5b4813a3a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4813de10 con 0x7f5b4809dc20 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5b481356d0 con 0x7f5b480a0a90 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5b4813df90 con 0x7f5b4813a3a0 2024-09-11T05:14:18.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b501c1640 10 monclient(hunting): authenticate will time out at 1341.694188s 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4e75e640 1 -- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b4813a3a0 msgr2=0x7f5b4813a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4e75e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b4813a3a0 0x7f5b4813a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b481d53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b481d53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:39696/0 (socket says 172.21.15.138:39696) 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 -- 172.21.15.138:0/3786799616 learned_addr learned my addr 172.21.15.138:0/3786799616 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient(hunting): get_auth_request con 0x7f5b4809dc20 auth_method 0 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:18.269 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4d75c640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b480a0a90 0x7f5b481d5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient(hunting): handle_auth_done global_id 64377 payload 824 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 -- 172.21.15.138:0/3786799616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b4813a3a0 msgr2=0x7f5b4813a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b4813a3a0 0x7f5b4813a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 -- 172.21.15.138:0/3786799616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b480a0a90 msgr2=0x7f5b481d5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b480a0a90 0x7f5b481d5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient: _finish_hunting 0 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 monclient: found mon.noname-a 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4df5d640 1 -- 172.21.15.138:0/3786799616 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4811f4d0 con 0x7f5b4809dc20 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4d75c640 10 monclient: get_auth_request con 0x7f5b480a0a90 auth_method 0 2024-09-11T05:14:18.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.254+0000 7f5b4d75c640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b480a0a90 0x7f5b481d5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b4df5d640 1 -- 172.21.15.138:0/3786799616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 msgr2=0x7f5b481d53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b4df5d640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b481d53f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400b580 tx=0x7f5b4400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 client.?.objecter ms_handle_connect 0x7f5b4809dc20 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 1 -- 172.21.15.138:0/3786799616 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44017070 con 0x7f5b4809dc20 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: dump: 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:18.271 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: _finish_auth 0 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 5 monclient: authenticate success, global_id 64377 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 1 -- 172.21.15.138:0/3786799616 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b44003de0 con 0x7f5b4809dc20 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 10 monclient: _renew_subs 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4811f7b0 con 0x7f5b4809dc20 2024-09-11T05:14:18.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 1 -- 172.21.15.138:0/3786799616 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b440052f0 con 0x7f5b4809dc20 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 monclient: dump: 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:18.273 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3bfff640 4 set_mon_vals no callback set 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 10 client.64377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 10 monclient: _renew_subs 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b480a6300 con 0x7f5b4809dc20 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 1 librados: init done 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b4809e110 con 0x7f5b4809dc20 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 1 -- 172.21.15.138:0/3786799616 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5b44005510 con 0x7f5b4809dc20 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 1 --2- 172.21.15.138:0/3786799616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5b2c03ce50 0x7f5b2c03f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:18.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 1 -- 172.21.15.138:0/3786799616 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(318..318 src has 1..318) ==== 5374+0+0 (secure 0 0 0) 0x7f5b4404f0e0 con 0x7f5b4809dc20 2024-09-11T05:14:18.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 10 client.64377.objecter ms_dispatch 0x7f5b48000ba0 osd_map(318..318 src has 1..318) 2024-09-11T05:14:18.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 3 client.64377.objecter handle_osd_map got epochs [318,318] > 0 2024-09-11T05:14:18.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 3 client.64377.objecter handle_osd_map decoding full epoch 318 2024-09-11T05:14:18.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b3a7fc640 20 client.64377.objecter dump_active .. 0 homeless 2024-09-11T05:14:18.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b4d75c640 1 --2- 172.21.15.138:0/3786799616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5b2c03ce50 0x7f5b2c03f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:18.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.258+0000 7f5b4d75c640 10 monclient: get_auth_request con 0x7f5b2c03ce50 auth_method 0 2024-09-11T05:14:18.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.262+0000 7f5b4d75c640 1 --2- 172.21.15.138:0/3786799616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5b2c03ce50 0x7f5b2c03f310 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806ee80 tx=0x7f5b3c0a8000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:18.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.262+0000 7f5b3a7fc640 10 client.64377.objecter ms_handle_connect 0x7f5b2c03ce50 2024-09-11T05:14:18.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.262+0000 7f5b3a7fc640 1 -- 172.21.15.138:0/3786799616 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f5b44011480 con 0x7f5b4809dc20 2024-09-11T05:14:18.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.262+0000 7f5b3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:18.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.262+0000 7f5b3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:18.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.482+0000 7f5b501c1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:14:18.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.482+0000 7f5b501c1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:18.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.482+0000 7f5b501c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:18.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.482+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5b480a0e70 con 0x7f5b4809dc20 2024-09-11T05:14:18.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b3a7fc640 1 -- 172.21.15.138:0/3786799616 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f5b44011760 con 0x7f5b4809dc20 2024-09-11T05:14:18.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:18.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:18.488 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 10 librados: watch_flush enter 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 10 librados: watch_flush exit 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 20 client.64377.objecter shutdown clearing up homeless session... 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 10 client.64377.objecter successfully canceled tick 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5b2c03ce50 msgr2=0x7f5b2c03f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 --2- 172.21.15.138:0/3786799616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5b2c03ce50 0x7f5b2c03f310 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f5b4806ee80 tx=0x7f5b3c0a8000 comp rx=0 tx=0).stop 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 10 monclient: shutdown 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:18.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 msgr2=0x7f5b481d53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b481d53f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400b580 tx=0x7f5b4400ba40 comp rx=0 tx=0).stop 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 shutdown_connections 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 --2- 172.21.15.138:0/3786799616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5b2c03ce50 0x7f5b2c03f310 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5b4813a3a0 0x7f5b4813a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5b480a0a90 0x7f5b481d5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 --2- 172.21.15.138:0/3786799616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5b4809dc20 0x7f5b481d53f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 >> 172.21.15.138:0/3786799616 conn(0x7f5b48063640 msgr2=0x7f5b48071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 shutdown_connections 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 -- 172.21.15.138:0/3786799616 wait complete. 2024-09-11T05:14:18.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:18.486+0000 7f5b501c1640 1 librados: shutdown 2024-09-11T05:14:21.508 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:14:21.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient: get_monmap_and_config 2024-09-11T05:14:21.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient: build_initial_monmap 2024-09-11T05:14:21.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient: monmap: 2024-09-11T05:14:21.637 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:21.637 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:21.637 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:21.639025+0000 2024-09-11T05:14:21.637 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:21.639025+0000 2024-09-11T05:14:21.637 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:21.638 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:21.638 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:21.638 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:21.638 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:21.638 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:21.638 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:21.638 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 1 Processor -- start 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 1 -- start start 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient: init 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e0406e0e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient(hunting): picked mon.noname-c con 0x7f3e0409eb30 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient(hunting): picked mon.noname-a con 0x7f3e0409ef40 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e0406e4c0 con 0x7f3e0406e0e0 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f3e0409f820 con 0x7f3e0409ef40 2024-09-11T05:14:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3e0409f540 con 0x7f3e0409eb30 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e0a45e640 10 monclient(hunting): authenticate will time out at 1345.075630s 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e089fb640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e089fb640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:39702/0 (socket says 172.21.15.138:39702) 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 learned_addr learned my addr 172.21.15.138:0/412078508 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e089fb640 10 monclient(hunting): get_auth_request con 0x7f3e0409ef40 auth_method 0 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e089fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e089fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.634+0000 7f3e089fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f3e0409eb30 auth_method 0 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.638+0000 7f3e03fff640 1 -- 172.21.15.138:0/412078508 >> v1:172.21.15.190:6789/0 conn(0x7f3e0406e0e0 legacy=0x7f3e0409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:21.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:21.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 10 monclient(hunting): get_auth_request con 0x7f3e0409ef40 auth_method 0 2024-09-11T05:14:21.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:21.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f3e0409eb30 auth_method 0 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:14:21.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:21.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:21.842+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:14:22.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:22.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 10 monclient(hunting): get_auth_request con 0x7f3e0409ef40 auth_method 0 2024-09-11T05:14:22.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:22.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:22.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:22.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:14:22.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:14:22.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:22.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:14:22.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:22.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f3e0409eb30 auth_method 0 2024-09-11T05:14:22.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:22.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:22.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:22.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:14:22.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:14:22.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:22.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:22.246+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:14:23.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:23.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 10 monclient(hunting): get_auth_request con 0x7f3e0409ef40 auth_method 0 2024-09-11T05:14:23.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:23.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:23.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:23.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:14:23.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:14:23.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:23.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:14:23.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:23.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f3e0409eb30 auth_method 0 2024-09-11T05:14:23.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:23.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:23.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:23.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:14:23.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:14:23.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:23.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:23.050+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:14:24.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 10 monclient(hunting): tick 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 monclient(hunting): continuing hunt 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 msgr2=0x7f3e0413a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.190:3300/0 conn(0x7f3e0409eb30 0x7f3e0413a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 -- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 msgr2=0x7f3e0413ce80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 --2- 172.21.15.138:0/412078508 >> v2:172.21.15.138:3300/0 conn(0x7f3e0409ef40 0x7f3e0413ce80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 -- 172.21.15.138:0/412078508 >> v1:172.21.15.190:6789/0 conn(0x7f3e0406e0e0 legacy=0x7f3e0409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 10 monclient: _add_conns ranks=[0,2] 2024-09-11T05:14:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3de4003720 addr v1:172.21.15.138:6789/0 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 0x7f3de4009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 10 monclient(hunting): picked mon.noname-b con 0x7f3de4007350 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 -- 172.21.15.138:0/412078508 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e0409fb00 con 0x7f3de4003720 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 1 -- 172.21.15.138:0/412078508 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0409f820 con 0x7f3de4007350 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e027fc640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 msgr2=0x7f3de4009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 0x7f3de4009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.638+0000 7f3e03fff640 1 -- 172.21.15.138:0/412078508 >> v1:172.21.15.138:6789/0 conn(0x7f3de4003720 legacy=0x7f3de4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.842+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 msgr2=0x7f3de4009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:24.842+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 0x7f3de4009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:14:25.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.246+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 msgr2=0x7f3de4009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:25.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.246+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 0x7f3de4009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:14:25.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.534+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538228678 0 0) 0x7f3e0409fb00 con 0x7f3de4003720 2024-09-11T05:14:25.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.534+0000 7f3e037fe640 10 monclient(hunting): _negotiate 2024-09-11T05:14:25.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.534+0000 7f3e037fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.534+0000 7f3e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:25.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.534+0000 7f3e037fe640 10 monclient(hunting): my global_id is 64380 2024-09-11T05:14:25.539 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.534+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3df0003530 con 0x7f3de4003720 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1369155280 0 0) 0x7f3df0003530 con 0x7f3de4003720 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 msgr2=0x7f3de4009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3de4007350 0x7f3de4009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: _finish_hunting 0 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de400b050 con 0x7f3de4003720 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: _finish_auth 0 2024-09-11T05:14:25.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: _check_auth_tickets 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f3df4001a00 con 0x7f3de4003720 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: dump: 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 >> v1:172.21.15.138:6789/0 conn(0x7f3de4003720 legacy=0x7f3de4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 reap_dead start 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df00039c0 0x7f3df0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient(hunting): picked mon.b con 0x7f3df00039c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df0004810 0x7f3df0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient(hunting): picked mon.a con 0x7f3df0004810 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0005380 0x7f3df000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient(hunting): picked mon.c con 0x7f3df0005380 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0409f820 con 0x7f3df0004810 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3df000c2d0 con 0x7f3df00039c0 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3df000c4d0 con 0x7f3df0005380 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e089fb640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0005380 msgr2=0x7f3df000b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e037fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e089fb640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0005380 0x7f3df000b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df0004810 0x7f3df0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e03fff640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df00039c0 0x7f3df0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f3df0004810 auth_method 0 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e03fff640 10 monclient(hunting): get_auth_request con 0x7f3df00039c0 auth_method 0 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e03fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient(hunting): handle_auth_done global_id 64380 payload 293 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0005380 msgr2=0x7f3df000b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0005380 0x7f3df000b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.547 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df00039c0 msgr2=0x7f3df0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df00039c0 0x7f3df0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 1 monclient: found mon.a 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df0003530 con 0x7f3df0004810 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient: _finish_auth 0 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3dfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e0a45e640 5 monclient: authenticate success, global_id 64380 2024-09-11T05:14:25.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.538+0000 7f3e0a45e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:14:25.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.546+0000 7f3dfb7fe640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df0004810 msgr2=0x7f3df0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:25.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.546+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df0004810 0x7f3df0004c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00eae0 tx=0x7f3dec01d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af80c9b822763866 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec0b3070 con 0x7f3df0004810 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 10 monclient: dump: 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.554 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e0a45e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dec0ae410 con 0x7f3df0004810 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 1 -- 172.21.15.138:0/412078508 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec00f040 con 0x7f3df0004810 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e037fe640 10 monclient: dump: 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.555 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.556 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.556 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.556 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.556 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.556 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e0a45e640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:25.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e0a45e640 4 set_mon_vals no callback set 2024-09-11T05:14:25.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e0a45e640 10 monclient: shutdown 2024-09-11T05:14:25.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e0a45e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e0a45e640 1 -- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df0004810 msgr2=0x7f3df0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.550+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df0004810 0x7f3df0004c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00eae0 tx=0x7f3dec01d380 comp rx=0 tx=0).stop 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 -- 172.21.15.138:0/412078508 shutdown_connections 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0005380 0x7f3df000b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df0004810 0x7f3df0004c90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/412078508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df00039c0 0x7f3df0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 -- 172.21.15.138:0/412078508 >> 172.21.15.138:0/412078508 conn(0x7f3e04063640 msgr2=0x7f3e04063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 -- 172.21.15.138:0/412078508 shutdown_connections 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 -- 172.21.15.138:0/412078508 wait complete. 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient: build_initial_monmap 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient: monmap: 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:25.558 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 librados: starting msgr at 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 librados: starting objecter 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 Processor -- start 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 -- start start 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 librados: setting wanted keys 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 librados: calling monclient init 2024-09-11T05:14:25.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient: init 2024-09-11T05:14:25.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:25.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:14:25.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df00039c0 0x7f3e04126370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient(hunting): picked mon.noname-a con 0x7f3df00039c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:25.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0004810 0x7f3e04123db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient(hunting): picked mon.noname-c con 0x7f3df0004810 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df0005380 0x7f3e041242f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient(hunting): picked mon.noname-b con 0x7f3df0005380 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0409e9b0 con 0x7f3df00039c0 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0409e830 con 0x7f3df0005380 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3e04135f00 con 0x7f3df0004810 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e0a45e640 10 monclient(hunting): authenticate will time out at 1348.995334s 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3dfb7fe640 1 -- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0004810 msgr2=0x7f3e04123db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3dfb7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0004810 0x7f3e04123db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:25.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df00039c0 0x7f3e04126370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df00039c0 0x7f3e04126370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:39740/0 (socket says 172.21.15.138:39740) 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 1 -- 172.21.15.138:0/3307495507 learned_addr learned my addr 172.21.15.138:0/3307495507 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 10 monclient(hunting): get_auth_request con 0x7f3df00039c0 auth_method 0 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e089fb640 1 --2- 172.21.15.138:0/3307495507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df0005380 0x7f3e041242f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e089fb640 10 monclient(hunting): get_auth_request con 0x7f3df0005380 auth_method 0 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e089fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e089fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.554+0000 7f3e089fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:25.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e089fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e089fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e089fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 10 monclient(hunting): handle_auth_done global_id 64383 payload 824 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 1 -- 172.21.15.138:0/3307495507 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0004810 msgr2=0x7f3e04123db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 1 --2- 172.21.15.138:0/3307495507 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0004810 0x7f3e04123db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 1 -- 172.21.15.138:0/3307495507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df0005380 msgr2=0x7f3e041242f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 1 --2- 172.21.15.138:0/3307495507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df0005380 0x7f3e041242f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 10 monclient: _finish_hunting 0 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 1 monclient: found mon.noname-a 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 1 -- 172.21.15.138:0/3307495507 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e04127b70 con 0x7f3df00039c0 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 1 -- 172.21.15.138:0/3307495507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df00039c0 msgr2=0x7f3e04126370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:25.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e03fff640 1 --2- 172.21.15.138:0/3307495507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df00039c0 0x7f3e04126370 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3df4001b70 tx=0x7f3df4007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 client.?.objecter ms_handle_connect 0x7f3df00039c0 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 1 -- 172.21.15.138:0/3307495507 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4017020 con 0x7f3df00039c0 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: dump: 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.568 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: _finish_auth 0 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: _check_auth_tickets 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 5 monclient: authenticate success, global_id 64383 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 10 monclient: _renew_subs 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e0409d6f0 con 0x7f3df00039c0 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 1 -- 172.21.15.138:0/3307495507 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df400f910 con 0x7f3df00039c0 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 1 -- 172.21.15.138:0/3307495507 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df400fd00 con 0x7f3df00039c0 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:25.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e017fa640 10 monclient: dump: 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e02ffd640 4 set_mon_vals no callback set 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 10 client.64383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 10 monclient: _renew_subs 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e04126d00 con 0x7f3df00039c0 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 1 librados: init done 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.558+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e04136290 con 0x7f3df00039c0 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3e017fa640 1 -- 172.21.15.138:0/3307495507 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3df401c400 con 0x7f3df00039c0 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3e017fa640 1 --2- 172.21.15.138:0/3307495507 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3de403d730 0x7f3de403fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3e017fa640 1 -- 172.21.15.138:0/3307495507 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(318..318 src has 1..318) ==== 5374+0+0 (secure 0 0 0) 0x7f3df404fd80 con 0x7f3df00039c0 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3e017fa640 10 client.64383.objecter ms_dispatch 0x7f3e04000ba0 osd_map(318..318 src has 1..318) 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3e017fa640 3 client.64383.objecter handle_osd_map got epochs [318,318] > 0 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3e017fa640 3 client.64383.objecter handle_osd_map decoding full epoch 318 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3e017fa640 20 client.64383.objecter dump_active .. 0 homeless 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/3307495507 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3de403d730 0x7f3de403fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3dfb7fe640 10 monclient: get_auth_request con 0x7f3de403d730 auth_method 0 2024-09-11T05:14:25.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3e017fa640 10 client.64383.objecter ms_handle_connect 0x7f3de403d730 2024-09-11T05:14:25.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.562+0000 7f3dfb7fe640 1 --2- 172.21.15.138:0/3307495507 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3de403d730 0x7f3de403fbf0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f3dec000ee0 tx=0x7f3dec010040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.566+0000 7f3e017fa640 1 -- 172.21.15.138:0/3307495507 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3df4021020 con 0x7f3df00039c0 2024-09-11T05:14:25.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.566+0000 7f3e017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:25.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.566+0000 7f3e017fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:25.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.798+0000 7f3e0a45e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:14:25.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.798+0000 7f3e0a45e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:25.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.798+0000 7f3e0a45e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.798+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3e0409dea0 con 0x7f3df00039c0 2024-09-11T05:14:25.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.798+0000 7f3e017fa640 1 -- 172.21.15.138:0/3307495507 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f3df4023d10 con 0x7f3df00039c0 2024-09-11T05:14:25.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.798+0000 7f3e017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:14:25.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.798+0000 7f3e017fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:25.802 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:14:25.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 10 librados: watch_flush enter 2024-09-11T05:14:25.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 10 librados: watch_flush exit 2024-09-11T05:14:25.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 20 client.64383.objecter shutdown clearing up homeless session... 2024-09-11T05:14:25.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 10 client.64383.objecter successfully canceled tick 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3de403d730 msgr2=0x7f3de403fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/3307495507 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3de403d730 0x7f3de403fbf0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f3dec000ee0 tx=0x7f3dec010040 comp rx=0 tx=0).stop 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 10 monclient: shutdown 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df00039c0 msgr2=0x7f3e04126370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/3307495507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df00039c0 0x7f3e04126370 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3df4001b70 tx=0x7f3df4007410 comp rx=0 tx=0).stop 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 shutdown_connections 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/3307495507 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3de403d730 0x7f3de403fbf0 unknown :-1 s=CLOSED pgs=3410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/3307495507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3df0005380 0x7f3e041242f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/3307495507 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3df0004810 0x7f3e04123db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 --2- 172.21.15.138:0/3307495507 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3df00039c0 0x7f3e04126370 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 >> 172.21.15.138:0/3307495507 conn(0x7f3e04063640 msgr2=0x7f3e040a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:25.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 shutdown_connections 2024-09-11T05:14:25.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 -- 172.21.15.138:0/3307495507 wait complete. 2024-09-11T05:14:25.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.802+0000 7f3e0a45e640 1 librados: shutdown 2024-09-11T05:14:25.820 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:14:25.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T05:14:25.820 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:14:25.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient: get_monmap_and_config 2024-09-11T05:14:25.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient: build_initial_monmap 2024-09-11T05:14:25.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient: monmap: 2024-09-11T05:14:25.928 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:25.928 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.928 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:25.930047+0000 2024-09-11T05:14:25.928 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:25.930047+0000 2024-09-11T05:14:25.929 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:25.929 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.929 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:25.929 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:25.929 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:25.929 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:25.929 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:25.929 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 1 Processor -- start 2024-09-11T05:14:25.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 1 -- start start 2024-09-11T05:14:25.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient: init 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8640a32c0 0x7fc86409e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient(hunting): picked mon.noname-b con 0x7fc8640a32c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc86409e960 addr v1:172.21.15.138:6789/0 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc86409ed80 0x7fc864135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient(hunting): picked mon.noname-c con 0x7fc86409ed80 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86409f230 con 0x7fc86409e960 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc8640a1920 con 0x7fc86409ed80 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc8640a36a0 con 0x7fc8640a32c0 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc868a52640 10 monclient(hunting): authenticate will time out at 1349.366593s 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc86259c640 1 -- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8640a32c0 msgr2=0x7fc86409e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc86259c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8640a32c0 0x7fc86409e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc861d9b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc86409e960 0x7fc8641356f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50380/0 (socket says 172.21.15.138:50380) 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc861d9b640 1 -- 172.21.15.138:0/152129596 learned_addr learned my addr 172.21.15.138:0/152129596 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 1 --2- 172.21.15.138:0/152129596 >> v2:172.21.15.190:3300/0 conn(0x7fc86409ed80 0x7fc864135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient(hunting): get_auth_request con 0x7fc86409ed80 auth_method 0 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient(hunting): handle_auth_done global_id 74131 payload 274 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 1 -- 172.21.15.138:0/152129596 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8640a32c0 msgr2=0x7fc86409e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc8640a32c0 0x7fc86409e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 1 -- 172.21.15.138:0/152129596 >> v1:172.21.15.138:6789/0 conn(0x7fc86409e960 legacy=0x7fc8641356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 1 monclient: found mon.noname-c 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:14:25.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.926+0000 7fc862d9d640 1 -- 172.21.15.138:0/152129596 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8640a1b50 con 0x7fc86409ed80 2024-09-11T05:14:25.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc862d9d640 1 -- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 msgr2=0x7fc864135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc862d9d640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 0x7fc864135e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc8540097c0 tx=0x7fc85401c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e05472720f08ad6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc862d9d640 1 -- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 msgr2=0x7fc864135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc862d9d640 1 -- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 msgr2=0x7fc864135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc862d9d640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 0x7fc864135e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc8540097c0 tx=0x7fc85401c350 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc862d9d640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 0x7fc864135e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc8540097c0 tx=0x7fc85401c350 comp rx=0 tx=0).stop 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 10 monclient: ms_handle_reset current mon [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 1 -- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 msgr2=0x7fc864135e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 0x7fc864135e00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 10 monclient: _add_conns ranks=[3,1] 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 1 --2- 172.21.15.138:0/152129596 >> v2:172.21.15.138:3300/0 conn(0x7fc84c003510 0x7fc84c005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 10 monclient(hunting): picked mon.noname-a con 0x7fc84c003510 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:25.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc84c0062c0 addr v1:172.21.15.190:6789/0 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 1 -- 172.21.15.138:0/152129596 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84c00a230 con 0x7fc84c0062c0 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 1 -- 172.21.15.138:0/152129596 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fc84c00b430 con 0x7fc84c003510 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86159a640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 1 --2- 172.21.15.138:0/152129596 >> v2:172.21.15.138:3300/0 conn(0x7fc84c003510 0x7fc84c005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient(hunting): get_auth_request con 0x7fc84c003510 auth_method 0 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:14:25.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient(hunting): handle_auth_done global_id 74131 payload 293 2024-09-11T05:14:25.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 1 -- 172.21.15.138:0/152129596 >> v1:172.21.15.190:6789/0 conn(0x7fc84c0062c0 legacy=0x7fc84c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:25.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient: _finish_hunting 0 2024-09-11T05:14:25.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 1 monclient: found mon.noname-a 2024-09-11T05:14:25.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:25.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.934+0000 7fc86259c640 1 -- 172.21.15.138:0/152129596 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c00b6a0 con 0x7fc84c003510 2024-09-11T05:14:25.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86259c640 1 -- 172.21.15.138:0/152129596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc84c003510 msgr2=0x7fc84c005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:25.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86259c640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc84c003510 0x7fc84c005990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc85c003d10 tx=0x7fc854009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c345cd05aa1a6ce1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 1 -- 172.21.15.138:0/152129596 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c009a30 con 0x7fc84c003510 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: dump: 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.947 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: _finish_auth 0 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: _check_auth_tickets 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 5 monclient: authenticate success, global_id 74131 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 1 -- 172.21.15.138:0/152129596 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c015620 con 0x7fc84c003510 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 1 -- 172.21.15.138:0/152129596 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c015a10 con 0x7fc84c003510 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86159a640 10 monclient: dump: 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.948 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 4 set_mon_vals no callback set 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient: shutdown 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- 172.21.15.138:0/152129596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc84c003510 msgr2=0x7fc84c005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc84c003510 0x7fc84c005990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc85c003d10 tx=0x7fc854009190 comp rx=0 tx=0).stop 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 -- 172.21.15.138:0/152129596 reap_dead start 2024-09-11T05:14:25.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- 172.21.15.138:0/152129596 shutdown_connections 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc84c003510 0x7fc84c005990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 --2- 172.21.15.138:0/152129596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc86409ed80 0x7fc864135e00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- 172.21.15.138:0/152129596 >> 172.21.15.138:0/152129596 conn(0x7fc864063640 msgr2=0x7fc864063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- 172.21.15.138:0/152129596 shutdown_connections 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- 172.21.15.138:0/152129596 wait complete. 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient: build_initial_monmap 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient: monmap: 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.950 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 librados: starting msgr at 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 librados: starting objecter 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 Processor -- start 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- start start 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 librados: setting wanted keys 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 librados: calling monclient init 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient: init 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:14:25.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc86409ed80 0x7fc8641c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient(hunting): picked mon.noname-c con 0x7fc86409ed80 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc84c003510 0x7fc864135160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient(hunting): picked mon.noname-b con 0x7fc84c003510 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8641356a0 0x7fc86413b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient(hunting): picked mon.noname-a con 0x7fc8641356a0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc8640a12d0 con 0x7fc8641356a0 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc86406e1f0 con 0x7fc84c003510 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc86409e180 con 0x7fc86409ed80 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:25.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86259c640 1 -- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc86409ed80 msgr2=0x7fc8641c4e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc868a52640 10 monclient(hunting): authenticate will time out at 1349.378415s 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc86259c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc86409ed80 0x7fc8641c4e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8641356a0 0x7fc86413b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc861d9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc84c003510 0x7fc864135160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8641356a0 0x7fc86413b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:39752/0 (socket says 172.21.15.138:39752) 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 -- 172.21.15.138:0/3437117038 learned_addr learned my addr 172.21.15.138:0/3437117038 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient(hunting): get_auth_request con 0x7fc8641356a0 auth_method 0 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc861d9b640 10 monclient(hunting): get_auth_request con 0x7fc84c003510 auth_method 0 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc861d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc861d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc861d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc861d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc861d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc861d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient(hunting): handle_auth_done global_id 64386 payload 824 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc86409ed80 msgr2=0x7fc8641c4e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc86409ed80 0x7fc8641c4e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc84c003510 msgr2=0x7fc864135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc84c003510 0x7fc864135160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:14:25.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 monclient: found mon.noname-a 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.938+0000 7fc862d9d640 1 -- 172.21.15.138:0/3437117038 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc864135f00 con 0x7fc8641356a0 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc862d9d640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8641356a0 msgr2=0x7fc86413b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc862d9d640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8641356a0 0x7fc86413b1d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc854000c10 tx=0x7fc8540190f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 client.?.objecter ms_handle_connect 0x7fc8641356a0 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540b0070 con 0x7fc8641356a0 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: dump: 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.955 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: _finish_auth 0 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8540b6420 con 0x7fc8641356a0 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540b6810 con 0x7fc8641356a0 2024-09-11T05:14:25.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 5 monclient: authenticate success, global_id 64386 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 10 monclient: _renew_subs 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86411d890 con 0x7fc8641356a0 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc860d99640 4 set_mon_vals no callback set 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 monclient: dump: 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.957 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 10 client.64386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 10 monclient: _renew_subs 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc86411f350 con 0x7fc8641356a0 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 1 librados: init done 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86411f520 0x7fc86413b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc8640a1450 con 0x7fc86411f520 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86411f520 0x7fc86413b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 10 monclient: get_auth_request con 0x7fc86411f520 auth_method 0 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fc8540b69b0 con 0x7fc8641356a0 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 10 monclient(hunting): handle_auth_done global_id 64389 payload 824 2024-09-11T05:14:25.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 1 --2- 172.21.15.138:0/3437117038 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc84403ced0 0x7fc84403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc86259c640 1 -- 172.21.15.138:0/3437117038 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc86409e420 con 0x7fc86411f520 2024-09-11T05:14:25.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(318..318 src has 1..318) ==== 5374+0+0 (secure 0 0 0) 0x7fc8540e9b40 con 0x7fc8641356a0 2024-09-11T05:14:25.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 10 client.64386.objecter ms_dispatch 0x7fc864000ba0 osd_map(318..318 src has 1..318) 2024-09-11T05:14:25.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 3 client.64386.objecter handle_osd_map got epochs [318,318] > 0 2024-09-11T05:14:25.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 3 client.64386.objecter handle_osd_map decoding full epoch 318 2024-09-11T05:14:25.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc852ffd640 20 client.64386.objecter dump_active .. 0 homeless 2024-09-11T05:14:25.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc861d9b640 1 --2- 172.21.15.138:0/3437117038 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc84403ced0 0x7fc84403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.942+0000 7fc861d9b640 10 monclient: get_auth_request con 0x7fc84403ced0 auth_method 0 2024-09-11T05:14:25.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 10 client.64386.objecter ms_handle_connect 0x7fc86411f520 2024-09-11T05:14:25.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc86259c640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86411f520 msgr2=0x7fc86413b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:25.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 10 client.64386.objecter resend_mon_ops 2024-09-11T05:14:25.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc86259c640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86411f520 0x7fc86413b710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc85c003d10 tx=0x7fc85c005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c015400 con 0x7fc86411f520 2024-09-11T05:14:25.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 10 monclient: dump: 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc85c015400 con 0x7fc86411f520 2024-09-11T05:14:25.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86411f520 msgr2=0x7fc86413b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86411f520 0x7fc86413b710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc85c003d10 tx=0x7fc85c005da0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86411f520 msgr2=0x7fc86413b710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86411f520 0x7fc86413b710 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc852ffd640 10 client.64386.objecter ms_handle_connect 0x7fc84403ced0 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.946+0000 7fc861d9b640 1 --2- 172.21.15.138:0/3437117038 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc84403ced0 0x7fc84403f390 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fc858005fd0 tx=0x7fc858007bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc868a52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:14:25.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc868a52640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86413bc50 0x7fc8640a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc868a52640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc86413c060 con 0x7fc86413bc50 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86413bc50 0x7fc8640a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 10 monclient: get_auth_request con 0x7fc86413bc50 auth_method 0 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 10 monclient(hunting): handle_auth_done global_id 64392 payload 824 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.950+0000 7fc86259c640 1 -- 172.21.15.138:0/3437117038 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc86406cca0 con 0x7fc86413bc50 2024-09-11T05:14:25.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc86259c640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86413bc50 msgr2=0x7fc8640a21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc86259c640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86413bc50 0x7fc8640a21b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc85c000f90 tx=0x7fc85c009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 10 client.64386.objecter ms_handle_connect 0x7fc86413bc50 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 10 client.64386.objecter resend_mon_ops 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c020600 con 0x7fc86413bc50 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 10 monclient: dump: 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:25.967 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fc85c00d3f0 con 0x7fc86413bc50 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86413bc50 msgr2=0x7fc8640a21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86413bc50 0x7fc8640a21b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc85c000f90 tx=0x7fc85c009520 comp rx=0 tx=0).stop 2024-09-11T05:14:25.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86413bc50 msgr2=0x7fc8640a21b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:25.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc852ffd640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc86413bc50 0x7fc8640a21b0 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc85c000f90 tx=0x7fc85c009520 comp rx=0 tx=0).stop 2024-09-11T05:14:25.969 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:14:25.969 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:25.969 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:25.969 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:14:25.969 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 30, 2024-09-11T05:14:25.969 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 169584, 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: 1 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:25.970 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:25.971 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:25.972 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:14:25.973 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:25.974 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:25.975 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:25.976 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:14:25.977 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:25.978 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:14:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:25.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:25.982 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:14:25.982 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 10 librados: watch_flush enter 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 10 librados: watch_flush exit 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 20 client.64386.objecter shutdown clearing up homeless session... 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 10 client.64386.objecter successfully canceled tick 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc84403ced0 msgr2=0x7fc84403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 --2- 172.21.15.138:0/3437117038 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc84403ced0 0x7fc84403f390 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fc858005fd0 tx=0x7fc858007bf0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 10 monclient: shutdown 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8641356a0 msgr2=0x7fc86413b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 --2- 172.21.15.138:0/3437117038 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc8641356a0 0x7fc86413b1d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc854000c10 tx=0x7fc8540190f0 comp rx=0 tx=0).stop 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc862d9d640 1 -- 172.21.15.138:0/3437117038 reap_dead start 2024-09-11T05:14:25.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc862d9d640 1 -- 172.21.15.138:0/3437117038 reap_dead start 2024-09-11T05:14:25.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 shutdown_connections 2024-09-11T05:14:25.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 >> 172.21.15.138:0/3437117038 conn(0x7fc864063640 msgr2=0x7fc864072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:25.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 shutdown_connections 2024-09-11T05:14:25.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 -- 172.21.15.138:0/3437117038 wait complete. 2024-09-11T05:14:25.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:25.954+0000 7fc868a52640 1 librados: shutdown 2024-09-11T05:14:25.985 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:14:25.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T05:14:25.985 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:14:26.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient: get_monmap_and_config 2024-09-11T05:14:26.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient: build_initial_monmap 2024-09-11T05:14:26.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient: monmap: 2024-09-11T05:14:26.088 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:26.088 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:26.088 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:14:26.090140+0000 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:14:26.090140+0000 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:14:26.089 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:26.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 1 Processor -- start 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 1 -- start start 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient: init 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f233809da20 0x7f23380a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient(hunting): picked mon.noname-c con 0x7f233809da20 addr v2:172.21.15.190:3300/0 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f23380a1e60 0x7f23380a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient(hunting): picked mon.noname-a con 0x7f23380a1e60 addr v2:172.21.15.138:3300/0 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2338136200 addr v1:172.21.15.138:6789/0 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2338064590 con 0x7f2338136200 2024-09-11T05:14:26.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f233809de00 con 0x7f23380a1e60 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f23380a2c70 con 0x7f233809da20 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233eb2f640 10 monclient(hunting): authenticate will time out at 1349.526355s 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f23380a1e60 0x7f23380a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233d0cc640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f2338136200 0x7f2338136610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50384/0 (socket says 172.21.15.138:50384) 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233d0cc640 1 -- 172.21.15.138:0/4100648763 learned_addr learned my addr 172.21.15.138:0/4100648763 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient(hunting): get_auth_request con 0x7f23380a1e60 auth_method 0 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:26.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233c8cb640 1 --2- 172.21.15.138:0/4100648763 >> v2:172.21.15.190:3300/0 conn(0x7f233809da20 0x7f23380a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233c8cb640 10 monclient(hunting): get_auth_request con 0x7f233809da20 auth_method 0 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233c8cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233c8cb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f233c8cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient(hunting): handle_auth_done global_id 64395 payload 274 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 1 -- 172.21.15.138:0/4100648763 >> v2:172.21.15.190:3300/0 conn(0x7f233809da20 msgr2=0x7f23380a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 1 --2- 172.21.15.138:0/4100648763 >> v2:172.21.15.190:3300/0 conn(0x7f233809da20 0x7f23380a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 1 -- 172.21.15.138:0/4100648763 >> v1:172.21.15.138:6789/0 conn(0x7f2338136200 legacy=0x7f2338136610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient: _finish_hunting 0 2024-09-11T05:14:26.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 1 monclient: found mon.noname-a 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.086+0000 7f2337fff640 1 -- 172.21.15.138:0/4100648763 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23380a2f50 con 0x7f23380a1e60 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f2337fff640 1 -- 172.21.15.138:0/4100648763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23380a1e60 msgr2=0x7f23380a22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f2337fff640 1 --2- 172.21.15.138:0/4100648763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23380a1e60 0x7f23380a22d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f233806d670 tx=0x7f232801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a0a5f86360312b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 1 -- 172.21.15.138:0/4100648763 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23280b0070 con 0x7f23380a1e60 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: dump: 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.098 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: _finish_auth 0 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: _check_auth_tickets 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f233eb2f640 5 monclient: authenticate success, global_id 64395 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f233eb2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 1 -- 172.21.15.138:0/4100648763 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23280ab600 con 0x7f23380a1e60 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 1 -- 172.21.15.138:0/4100648763 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23280ab9d0 con 0x7f23380a1e60 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:26.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f23377fe640 10 monclient: dump: 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f233eb2f640 10 monclient: get_monmap_and_config success 2024-09-11T05:14:26.100 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.090+0000 7f233eb2f640 4 set_mon_vals no callback set 2024-09-11T05:14:26.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient: shutdown 2024-09-11T05:14:26.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- 172.21.15.138:0/4100648763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23380a1e60 msgr2=0x7f23380a22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 --2- 172.21.15.138:0/4100648763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23380a1e60 0x7f23380a22d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f233806d670 tx=0x7f232801c340 comp rx=0 tx=0).stop 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- 172.21.15.138:0/4100648763 shutdown_connections 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 --2- 172.21.15.138:0/4100648763 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f23380a1e60 0x7f23380a22d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 --2- 172.21.15.138:0/4100648763 >> v2:172.21.15.190:3300/0 conn(0x7f233809da20 0x7f23380a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- 172.21.15.138:0/4100648763 >> 172.21.15.138:0/4100648763 conn(0x7f2338063640 msgr2=0x7f2338063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- 172.21.15.138:0/4100648763 shutdown_connections 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- 172.21.15.138:0/4100648763 wait complete. 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient: build_initial_monmap 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient: monmap: 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:14:26.107 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 librados: starting msgr at 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 librados: starting objecter 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 Processor -- start 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- start start 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 librados: setting wanted keys 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 librados: calling monclient init 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient: init 2024-09-11T05:14:26.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient: _reopen_session rank -1 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f233809da20 0x7f2338137620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient(hunting): picked mon.noname-a con 0x7f233809da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380a1e60 0x7f2338137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient(hunting): picked mon.noname-b con 0x7f23380a1e60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2338136200 0x7f23381380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient(hunting): picked mon.noname-c con 0x7f2338136200 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f23381410f0 con 0x7f233809da20 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2338140f70 con 0x7f23380a1e60 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:26.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2338141270 con 0x7f2338136200 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient(hunting): _renew_subs 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233eb2f640 10 monclient(hunting): authenticate will time out at 1349.533661s 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233d0cc640 1 -- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2338136200 msgr2=0x7f23381380a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3301/0 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233d0cc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2338136200 0x7f23381380a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f233809da20 0x7f2338137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f233809da20 0x7f2338137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:39784/0 (socket says 172.21.15.138:39784) 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 -- 172.21.15.138:0/3094952616 learned_addr learned my addr 172.21.15.138:0/3094952616 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient(hunting): get_auth_request con 0x7f233809da20 auth_method 0 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:26.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient(hunting): handle_auth_done global_id 64398 payload 824 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f2337fff640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380a1e60 0x7f2338137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2338136200 msgr2=0x7f23381380a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2338136200 0x7f23381380a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380a1e60 msgr2=0x7f2338137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380a1e60 0x7f2338137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient: _finish_hunting 0 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 monclient: found mon.noname-a 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.094+0000 7f233c8cb640 1 -- 172.21.15.138:0/3094952616 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f233813b2f0 con 0x7f233809da20 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233c8cb640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f233809da20 msgr2=0x7f2338137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233c8cb640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f233809da20 0x7f2338137620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f232c00d800 tx=0x7f232c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 client.?.objecter ms_handle_connect 0x7f233809da20 2024-09-11T05:14:26.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c00b8d0 con 0x7f233809da20 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: dump: 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:26.112 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: _finish_auth 0 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: _check_auth_tickets 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f232c019070 con 0x7f233809da20 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c00bc70 con 0x7f233809da20 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 monclient: dump: 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.113 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 5 monclient: authenticate success, global_id 64398 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 10 monclient: _renew_subs 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2336ffd640 4 set_mon_vals no callback set 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:26.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2338135c50 con 0x7f233809da20 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 10 client.64398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 10 monclient: _renew_subs 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f233813e130 con 0x7f233809da20 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 1 librados: init done 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ce280 0x7f23381ce660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f23380a0430 con 0x7f23381ce280 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f232c013070 con 0x7f233809da20 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 1 --2- 172.21.15.138:0/3094952616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f232403ce50 0x7f232403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ce280 0x7f23381ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:26.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 10 monclient: get_auth_request con 0x7f23381ce280 auth_method 0 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(318..318 src has 1..318) ==== 5374+0+0 (secure 0 0 0) 0x7f232c04fbb0 con 0x7f233809da20 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 10 client.64398.objecter ms_dispatch 0x7f2338000ba0 osd_map(318..318 src has 1..318) 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 3 client.64398.objecter handle_osd_map got epochs [318,318] > 0 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 3 client.64398.objecter handle_osd_map decoding full epoch 318 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f23357fa640 20 client.64398.objecter dump_active .. 0 homeless 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233d0cc640 1 --2- 172.21.15.138:0/3094952616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f232403ce50 0x7f232403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f233d0cc640 10 monclient: get_auth_request con 0x7f232403ce50 auth_method 0 2024-09-11T05:14:26.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 10 monclient(hunting): handle_auth_done global_id 74140 payload 824 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.098+0000 7f2337fff640 1 -- 172.21.15.138:0/3094952616 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23380a1920 con 0x7f23381ce280 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 10 client.64398.objecter ms_handle_connect 0x7f23381ce280 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 10 client.64398.objecter resend_mon_ops 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f2337fff640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ce280 msgr2=0x7f23381ce660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f2337fff640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ce280 0x7f23381ce660 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23280ad000 tx=0x7f232801ab80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 10 client.64398.objecter ms_handle_connect 0x7f232403ce50 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f233d0cc640 1 --2- 172.21.15.138:0/3094952616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f232403ce50 0x7f232403f310 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f2338139590 tx=0x7f2330005090 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23280b0050 con 0x7f23381ce280 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 10 monclient: dump: 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:26.117 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f23280b1040 con 0x7f23381ce280 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ce280 msgr2=0x7f23381ce660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ce280 0x7f23381ce660 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23280ad000 tx=0x7f232801ab80 comp rx=0 tx=0).stop 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ce280 msgr2=0x7f23381ce660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:26.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.102+0000 7f23357fa640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ce280 0x7f23381ce660 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23280ad000 tx=0x7f232801ab80 comp rx=0 tx=0).stop 2024-09-11T05:14:26.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f233eb2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:14:26.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f233eb2f640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ceba0 0x7f233806f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f233eb2f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2338139af0 con 0x7f23381ceba0 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f2337fff640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ceba0 0x7f233806f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f2337fff640 10 monclient: get_auth_request con 0x7f23381ceba0 auth_method 0 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f2337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f2337fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f2337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.106+0000 7f2337fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f2337fff640 10 monclient(hunting): handle_auth_done global_id 74143 payload 824 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f2337fff640 1 -- 172.21.15.138:0/3094952616 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f233806cca0 con 0x7f23381ceba0 2024-09-11T05:14:26.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 10 client.64398.objecter ms_handle_connect 0x7f23381ceba0 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 10 client.64398.objecter resend_mon_ops 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f2337fff640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ceba0 msgr2=0x7f233806f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f2337fff640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ceba0 0x7f233806f2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2328000c10 tx=0x7f23280a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23280bfb30 con 0x7f23381ceba0 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 10 monclient: dump: 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:14:26.122 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f23280b5400 con 0x7f23381ceba0 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ceba0 msgr2=0x7f233806f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ceba0 0x7f233806f2e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2328000c10 tx=0x7f23280a8000 comp rx=0 tx=0).stop 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ceba0 msgr2=0x7f233806f2e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:14:26.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.110+0000 7f23357fa640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23381ceba0 0x7f233806f2e0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2328000c10 tx=0x7f23280a8000 comp rx=0 tx=0).stop 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 30, 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 33667, 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: 1 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:26.124 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:26.125 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:26.126 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:26.127 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:14:26.128 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:14:26.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:26.130 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:14:26.132 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:14:26.133 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:14:26.134 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:26.135 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:14:26.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:14:26.137 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:14:26.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 10 librados: watch_flush enter 2024-09-11T05:14:26.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 10 librados: watch_flush exit 2024-09-11T05:14:26.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 20 client.64398.objecter shutdown clearing up homeless session... 2024-09-11T05:14:26.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 10 client.64398.objecter successfully canceled tick 2024-09-11T05:14:26.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f232403ce50 msgr2=0x7f232403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:26.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 --2- 172.21.15.138:0/3094952616 >> v2:172.21.15.190:6800/1562782189 conn(0x7f232403ce50 0x7f232403f310 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f2338139590 tx=0x7f2330005090 comp rx=0 tx=0).stop 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 10 monclient: shutdown 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f233809da20 msgr2=0x7f2338137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f233809da20 0x7f2338137620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f232c00d800 tx=0x7f232c00dcc0 comp rx=0 tx=0).stop 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233d0cc640 1 -- 172.21.15.138:0/3094952616 reap_dead start 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 shutdown_connections 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 --2- 172.21.15.138:0/3094952616 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f233809da20 0x7f2338137620 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 >> 172.21.15.138:0/3094952616 conn(0x7f2338063640 msgr2=0x7f2338135370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 shutdown_connections 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 -- 172.21.15.138:0/3094952616 wait complete. 2024-09-11T05:14:26.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:14:26.114+0000 7f233eb2f640 1 librados: shutdown 2024-09-11T05:14:26.141 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:14:26.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:15:56.144 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-11T05:15:56.144 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T05:15:56.145 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T05:15:56.149 INFO:tasks.ceph.mon.c:Started 2024-09-11T05:15:56.149 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:15:56.149 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:15:56.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient: get_monmap_and_config 2024-09-11T05:15:56.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient: build_initial_monmap 2024-09-11T05:15:56.268 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient: monmap: 2024-09-11T05:15:56.268 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:15:56.268 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:56.268 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:15:56.271258+0000 2024-09-11T05:15:56.268 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:15:56.271258+0000 2024-09-11T05:15:56.269 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:15:56.269 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:56.269 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:15:56.269 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:15:56.269 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:15:56.269 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:15:56.269 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:15:56.269 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 1 Processor -- start 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 1 -- start start 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient: init 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient: _reopen_session rank -1 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f64700a5cc0 0x7f64700a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient(hunting): picked mon.noname-c con 0x7f64700a5cc0 addr v2:172.21.15.190:3300/0 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64700a60a0 addr v1:172.21.15.138:6789/0 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f6470140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient(hunting): picked mon.noname-b con 0x7f647013dd30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:15:56.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64700a3310 con 0x7f64700a60a0 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f64700a6bc0 con 0x7f64700a5cc0 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f64700643a0 con 0x7f647013dd30 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient(hunting): _renew_subs 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f64785f4640 10 monclient(hunting): authenticate will time out at 1439.707037s 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6475b8f640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f64700a60a0 0x7f64700a64b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:48886/0 (socket says 172.21.15.138:48886) 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f6470140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6475b8f640 1 -- 172.21.15.138:0/2546857625 learned_addr learned my addr 172.21.15.138:0/2546857625 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 10 monclient(hunting): get_auth_request con 0x7f647013dd30 auth_method 0 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:15:56.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 1 -- 172.21.15.138:0/2546857625 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 msgr2=0x7f6470140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 1 -- 172.21.15.138:0/2546857625 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 msgr2=0x7f6470140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 1 --2- 172.21.15.138:0/2546857625 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f6470140120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476b91640 1 --2- 172.21.15.138:0/2546857625 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f6470140120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 1 --2- 172.21.15.138:0/2546857625 >> v2:172.21.15.190:3300/0 conn(0x7f64700a5cc0 0x7f64700a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 10 monclient(hunting): get_auth_request con 0x7f64700a5cc0 auth_method 0 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.265+0000 7f6476390640 10 monclient(hunting): handle_auth_done global_id 74146 payload 274 2024-09-11T05:15:56.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.269+0000 7f6476390640 1 -- 172.21.15.138:0/2546857625 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 msgr2=0x7f6470140120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:15:56.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.269+0000 7f6476390640 1 --2- 172.21.15.138:0/2546857625 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f6470140120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.269+0000 7f6476390640 1 -- 172.21.15.138:0/2546857625 >> v1:172.21.15.138:6789/0 conn(0x7f64700a60a0 legacy=0x7f64700a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:15:56.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.269+0000 7f6476390640 10 monclient: _finish_hunting 0 2024-09-11T05:15:56.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.269+0000 7f6476390640 1 monclient: found mon.noname-c 2024-09-11T05:15:56.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.269+0000 7f6476390640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:15:56.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.269+0000 7f6476390640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:15:56.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.269+0000 7f6476390640 1 -- 172.21.15.138:0/2546857625 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64700a6f50 con 0x7f64700a5cc0 2024-09-11T05:15:56.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 1 -- 172.21.15.138:0/2546857625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a5cc0 msgr2=0x7f64700a3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:15:56.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 1 --2- 172.21.15.138:0/2546857625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a5cc0 0x7f64700a3d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f646c0097c0 tx=0x7f646c01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dde7989da80e6bc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:15:56.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 1 -- 172.21.15.138:0/2546857625 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c0b0070 con 0x7f64700a5cc0 2024-09-11T05:15:56.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:15:56.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:15:56.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: dump: 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: _finish_auth 0 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: _check_auth_tickets 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:15:56.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 1 -- 172.21.15.138:0/2546857625 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f646c0ab600 con 0x7f64700a5cc0 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: handle_config config(0 keys) 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 1 -- 172.21.15.138:0/2546857625 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c0ab9d0 con 0x7f64700a5cc0 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f647538e640 10 monclient: dump: 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:15:56.279 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 5 monclient: authenticate success, global_id 74146 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient: get_monmap_and_config success 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 4 set_mon_vals no callback set 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient: shutdown 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- 172.21.15.138:0/2546857625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a5cc0 msgr2=0x7f64700a3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 --2- 172.21.15.138:0/2546857625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a5cc0 0x7f64700a3d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f646c0097c0 tx=0x7f646c01c360 comp rx=0 tx=0).stop 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- 172.21.15.138:0/2546857625 shutdown_connections 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 --2- 172.21.15.138:0/2546857625 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f6470140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 --2- 172.21.15.138:0/2546857625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a5cc0 0x7f64700a3d40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- 172.21.15.138:0/2546857625 >> 172.21.15.138:0/2546857625 conn(0x7f64700a8670 msgr2=0x7f6470063640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- 172.21.15.138:0/2546857625 shutdown_connections 2024-09-11T05:15:56.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- 172.21.15.138:0/2546857625 wait complete. 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient: build_initial_monmap 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient: monmap: 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 librados: starting msgr at 2024-09-11T05:15:56.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 librados: starting objecter 2024-09-11T05:15:56.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 Processor -- start 2024-09-11T05:15:56.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- start start 2024-09-11T05:15:56.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 librados: setting wanted keys 2024-09-11T05:15:56.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 librados: calling monclient init 2024-09-11T05:15:56.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient: init 2024-09-11T05:15:56.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient: _reopen_session rank -1 2024-09-11T05:15:56.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64700a5cc0 0x7f6470121eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient(hunting): picked mon.noname-a con 0x7f64700a5cc0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a60a0 0x7f6470138d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient(hunting): picked mon.noname-b con 0x7f64700a60a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f647009f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient(hunting): picked mon.noname-c con 0x7f647013dd30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6470143320 con 0x7f64700a5cc0 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f64701431a0 con 0x7f64700a60a0 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f64701434a0 con 0x7f647013dd30 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient(hunting): _renew_subs 2024-09-11T05:15:56.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f64785f4640 10 monclient(hunting): authenticate will time out at 1439.715822s 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64700a5cc0 0x7f6470121eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476b91640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f647009f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64700a5cc0 0x7f6470121eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:57230/0 (socket says 172.21.15.138:57230) 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 1 -- 172.21.15.138:0/1695159561 learned_addr learned my addr 172.21.15.138:0/1695159561 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 10 monclient(hunting): get_auth_request con 0x7f64700a5cc0 auth_method 0 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.273+0000 7f6476390640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476b91640 10 monclient(hunting): get_auth_request con 0x7f647013dd30 auth_method 0 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476b91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476b91640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476b91640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:15:56.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476b91640 1 -- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 msgr2=0x7f647009f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476b91640 1 -- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 msgr2=0x7f647009f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476b91640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f647009f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476b91640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f647009f9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6475b8f640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a60a0 0x7f6470138d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 10 monclient(hunting): handle_auth_done global_id 64401 payload 824 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 1 -- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 msgr2=0x7f647009f9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f647009f9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 1 -- 172.21.15.138:0/1695159561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a60a0 msgr2=0x7f6470138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a60a0 0x7f6470138d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 10 monclient: _finish_hunting 0 2024-09-11T05:15:56.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 1 monclient: found mon.noname-a 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 1 -- 172.21.15.138:0/1695159561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64701233d0 con 0x7f64700a5cc0 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6475b8f640 10 monclient: get_auth_request con 0x7f64700a60a0 auth_method 0 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6475b8f640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a60a0 0x7f6470138d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 1 -- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64700a5cc0 msgr2=0x7f6470121eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6476390640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64700a5cc0 0x7f6470121eb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f646c01cc00 tx=0x7f646c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 client.?.objecter ms_handle_connect 0x7f64700a5cc0 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 1 -- 172.21.15.138:0/1695159561 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c0b9070 con 0x7f64700a5cc0 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: dump: 2024-09-11T05:15:56.290 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: _finish_auth 0 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 1 -- 172.21.15.138:0/1695159561 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f646c0b5690 con 0x7f64700a5cc0 2024-09-11T05:15:56.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 1 -- 172.21.15.138:0/1695159561 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c0b0040 con 0x7f64700a5cc0 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 5 monclient: authenticate success, global_id 64401 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 10 monclient: _renew_subs 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64700aa360 con 0x7f64700a5cc0 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6474b8d640 4 set_mon_vals no callback set 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 10 monclient: dump: 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:56.292 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 10 client.64401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 10 monclient: _renew_subs 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64701ce8b0 con 0x7f64700a5cc0 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 1 librados: init done 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:15:56.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64700a2490 con 0x7f64700a5cc0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 1 -- 172.21.15.138:0/1695159561 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f646c0b5ac0 con 0x7f64700a5cc0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.277+0000 7f6466ffd640 1 --2- 172.21.15.138:0/1695159561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403cea0 0x7f645403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6466ffd640 1 -- 172.21.15.138:0/1695159561 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(359..359 src has 1..359) ==== 5390+0+0 (secure 0 0 0) 0x7f646c0ea010 con 0x7f64700a5cc0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6466ffd640 10 client.64401.objecter ms_dispatch 0x7f6470000ba0 osd_map(359..359 src has 1..359) 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6466ffd640 3 client.64401.objecter handle_osd_map got epochs [359,359] > 0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6466ffd640 3 client.64401.objecter handle_osd_map decoding full epoch 359 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6466ffd640 20 client.64401.objecter dump_active .. 0 homeless 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6475b8f640 1 --2- 172.21.15.138:0/1695159561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403cea0 0x7f645403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6475b8f640 10 monclient: get_auth_request con 0x7f645403cea0 auth_method 0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6475b8f640 1 --2- 172.21.15.138:0/1695159561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403cea0 0x7f645403f360 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7f6460007fd0 tx=0x7f6460009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.281+0000 7f6466ffd640 10 client.64401.objecter ms_handle_connect 0x7f645403cea0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.285+0000 7f6466ffd640 1 -- 172.21.15.138:0/1695159561 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f646c016070 con 0x7f64700a5cc0 2024-09-11T05:15:56.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.285+0000 7f6466ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:15:56.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.285+0000 7f6466ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:15:56.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.521+0000 7f64785f4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:15:56.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.521+0000 7f64785f4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:15:56.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.521+0000 7f64785f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:15:56.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.521+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f64701ceae0 con 0x7f64700a5cc0 2024-09-11T05:15:56.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.521+0000 7f6466ffd640 1 -- 172.21.15.138:0/1695159561 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f64701ceae0 con 0x7f64700a5cc0 2024-09-11T05:15:56.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.521+0000 7f6466ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:15:56.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.521+0000 7f6466ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:15:56.524 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:15:56.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 10 librados: watch_flush enter 2024-09-11T05:15:56.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 10 librados: watch_flush exit 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 20 client.64401.objecter shutdown clearing up homeless session... 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 10 client.64401.objecter successfully canceled tick 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403cea0 msgr2=0x7f645403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 --2- 172.21.15.138:0/1695159561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403cea0 0x7f645403f360 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7f6460007fd0 tx=0x7f6460009040 comp rx=0 tx=0).stop 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 10 monclient: shutdown 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64700a5cc0 msgr2=0x7f6470121eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64700a5cc0 0x7f6470121eb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f646c01cc00 tx=0x7f646c019100 comp rx=0 tx=0).stop 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 shutdown_connections 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 --2- 172.21.15.138:0/1695159561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403cea0 0x7f645403f360 unknown :-1 s=CLOSED pgs=3415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f647013dd30 0x7f647009f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f64700a60a0 0x7f6470138d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 --2- 172.21.15.138:0/1695159561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f64700a5cc0 0x7f6470121eb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:56.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 >> 172.21.15.138:0/1695159561 conn(0x7f64700a8670 msgr2=0x7f647013eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:15:56.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 shutdown_connections 2024-09-11T05:15:56.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 -- 172.21.15.138:0/1695159561 wait complete. 2024-09-11T05:15:56.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:56.525+0000 7f64785f4640 1 librados: shutdown 2024-09-11T05:15:59.542 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:15:59.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient: get_monmap_and_config 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient: build_initial_monmap 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient: monmap: 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:15:59.670866+0000 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:15:59.670866+0000 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:15:59.668 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:15:59.669 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:15:59.669 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:59.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 1 Processor -- start 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 1 -- start start 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient: init 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient: _reopen_session rank -1 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f341406e0e0 0x7f3414136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient(hunting): picked mon.noname-a con 0x7f341406e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3414136170 0x7f3414136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3414136170 addr v2:172.21.15.190:3300/0 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34140a1920 addr v1:172.21.15.138:6789/0 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34140a3270 con 0x7f34140a1920 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:59.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f341406e4c0 con 0x7f341406e0e0 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f34140a0af0 con 0x7f3414136170 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient(hunting): _renew_subs 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f341b9c6640 10 monclient(hunting): authenticate will time out at 1443.106857s 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f341406e0e0 0x7f3414136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419f63640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f34140a1920 0x7f34140a1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:60412/0 (socket says 172.21.15.138:60412) 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419f63640 1 -- 172.21.15.138:0/3967359788 learned_addr learned my addr 172.21.15.138:0/3967359788 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient(hunting): get_auth_request con 0x7f341406e0e0 auth_method 0 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:15:59.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient(hunting): handle_auth_done global_id 64404 payload 274 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 1 -- 172.21.15.138:0/3967359788 >> v2:172.21.15.190:3300/0 conn(0x7f3414136170 msgr2=0x7f3414136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3418f61640 1 --2- 172.21.15.138:0/3967359788 >> v2:172.21.15.190:3300/0 conn(0x7f3414136170 0x7f3414136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 1 --2- 172.21.15.138:0/3967359788 >> v2:172.21.15.190:3300/0 conn(0x7f3414136170 0x7f3414136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 1 -- 172.21.15.138:0/3967359788 >> v1:172.21.15.138:6789/0 conn(0x7f34140a1920 legacy=0x7f34140a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient: _finish_hunting 0 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 1 monclient: found mon.noname-a 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3419762640 1 -- 172.21.15.138:0/3967359788 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34140a2f90 con 0x7f341406e0e0 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3418f61640 10 monclient: get_auth_request con 0x7f3414136170 auth_method 0 2024-09-11T05:15:59.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.665+0000 7f3418f61640 1 --2- 172.21.15.138:0/3967359788 >> v2:172.21.15.190:3300/0 conn(0x7f3414136170 0x7f3414136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:15:59.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f3419762640 1 -- 172.21.15.138:0/3967359788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 msgr2=0x7f3414136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:15:59.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f3419762640 1 --2- 172.21.15.138:0/3967359788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f3414136720 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3414130830 tx=0x7f341001c160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd855a9645eaa6b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:15:59.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 1 -- 172.21.15.138:0/3967359788 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34100b1070 con 0x7f341406e0e0 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: dump: 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:15:59.676 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: _finish_auth 0 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 1 -- 172.21.15.138:0/3967359788 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34100ac440 con 0x7f341406e0e0 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 1 -- 172.21.15.138:0/3967359788 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34100ac810 con 0x7f341406e0e0 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f341b9c6640 5 monclient: authenticate success, global_id 64404 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f341b9c6640 10 monclient: get_monmap_and_config success 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f341b9c6640 4 set_mon_vals no callback set 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:15:59.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.669+0000 7f340bfff640 10 monclient: dump: 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient: shutdown 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- 172.21.15.138:0/3967359788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 msgr2=0x7f3414136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:15:59.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 --2- 172.21.15.138:0/3967359788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f3414136720 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3414130830 tx=0x7f341001c160 comp rx=0 tx=0).stop 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- 172.21.15.138:0/3967359788 shutdown_connections 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 --2- 172.21.15.138:0/3967359788 >> v2:172.21.15.190:3300/0 conn(0x7f3414136170 0x7f3414136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 --2- 172.21.15.138:0/3967359788 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f3414136720 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- 172.21.15.138:0/3967359788 >> 172.21.15.138:0/3967359788 conn(0x7f3414063640 msgr2=0x7f3414063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- 172.21.15.138:0/3967359788 shutdown_connections 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- 172.21.15.138:0/3967359788 wait complete. 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient: build_initial_monmap 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient: monmap: 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:15:59.679 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 librados: starting msgr at 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 librados: starting objecter 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 Processor -- start 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- start start 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 librados: setting wanted keys 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 librados: calling monclient init 2024-09-11T05:15:59.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient: init 2024-09-11T05:15:59.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient: _reopen_session rank -1 2024-09-11T05:15:59.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:15:59.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f34141ce910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:59.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient(hunting): picked mon.noname-a con 0x7f341406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:15:59.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34140a1920 0x7f341413d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient(hunting): picked mon.noname-b con 0x7f34140a1920 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 0x7f341413dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3414136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f34141410a0 con 0x7f341406e0e0 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3414140f20 con 0x7f34140a1920 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3414141220 con 0x7f3414136170 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient(hunting): _renew_subs 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f341b9c6640 10 monclient(hunting): authenticate will time out at 1443.114193s 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f34141ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:59.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f34141ce910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37936/0 (socket says 172.21.15.138:37936) 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 -- 172.21.15.138:0/1735370487 learned_addr learned my addr 172.21.15.138:0/1735370487 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient(hunting): get_auth_request con 0x7f341406e0e0 auth_method 0 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 0x7f341413dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 10 monclient(hunting): get_auth_request con 0x7f3414136170 auth_method 0 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 1 -- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 msgr2=0x7f341413dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:15:59.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 1 -- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 msgr2=0x7f341413dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 0x7f341413dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419f63640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 0x7f341413dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3418f61640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34140a1920 0x7f341413d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3418f61640 10 monclient(hunting): get_auth_request con 0x7f34140a1920 auth_method 0 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3418f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3418f61640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3418f61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient(hunting): handle_auth_done global_id 64407 payload 824 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 -- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 msgr2=0x7f341413dda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 0x7f341413dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 -- 172.21.15.138:0/1735370487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34140a1920 msgr2=0x7f341413d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34140a1920 0x7f341413d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient: _finish_hunting 0 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 monclient: found mon.noname-a 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.673+0000 7f3419762640 1 -- 172.21.15.138:0/1735370487 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3414138fe0 con 0x7f341406e0e0 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3419762640 1 -- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 msgr2=0x7f34141ce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3419762640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f34141ce910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f34100af040 tx=0x7f3410018f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 client.?.objecter ms_handle_connect 0x7f341406e0e0 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 1 -- 172.21.15.138:0/1735370487 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34100b1040 con 0x7f341406e0e0 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: dump: 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:15:59.688 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: _finish_auth 0 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 1 -- 172.21.15.138:0/1735370487 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34100ba070 con 0x7f341406e0e0 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 5 monclient: authenticate success, global_id 64407 2024-09-11T05:15:59.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 10 monclient: _renew_subs 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34141338e0 con 0x7f341406e0e0 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 1 -- 172.21.15.138:0/1735370487 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34100b4aa0 con 0x7f341406e0e0 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 monclient: dump: 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:15:59.690 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f340affd640 4 set_mon_vals no callback set 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 10 client.64407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 10 monclient: _renew_subs 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341413e2e0 con 0x7f341406e0e0 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 1 librados: init done 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34140a1d30 con 0x7f341406e0e0 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 1 -- 172.21.15.138:0/1735370487 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f34100b4050 con 0x7f341406e0e0 2024-09-11T05:15:59.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 1 --2- 172.21.15.138:0/1735370487 >> v2:172.21.15.190:6800/1562782189 conn(0x7f33f803cf20 0x7f33f803f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 1 -- 172.21.15.138:0/1735370487 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(359..359 src has 1..359) ==== 5390+0+0 (secure 0 0 0) 0x7f34100e9030 con 0x7f341406e0e0 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 10 client.64407.objecter ms_dispatch 0x7f3414000ba0 osd_map(359..359 src has 1..359) 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 3 client.64407.objecter handle_osd_map got epochs [359,359] > 0 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 3 client.64407.objecter handle_osd_map decoding full epoch 359 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3409ffb640 20 client.64407.objecter dump_active .. 0 homeless 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3418f61640 1 --2- 172.21.15.138:0/1735370487 >> v2:172.21.15.190:6800/1562782189 conn(0x7f33f803cf20 0x7f33f803f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.677+0000 7f3418f61640 10 monclient: get_auth_request con 0x7f33f803cf20 auth_method 0 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.681+0000 7f3418f61640 1 --2- 172.21.15.138:0/1735370487 >> v2:172.21.15.190:6800/1562782189 conn(0x7f33f803cf20 0x7f33f803f3e0 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 crypto rx=0x7f3414139dc0 tx=0x7f3404008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:15:59.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.681+0000 7f3409ffb640 10 client.64407.objecter ms_handle_connect 0x7f33f803cf20 2024-09-11T05:15:59.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.681+0000 7f3409ffb640 1 -- 172.21.15.138:0/1735370487 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f34100c3020 con 0x7f341406e0e0 2024-09-11T05:15:59.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.681+0000 7f3409ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:15:59.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.681+0000 7f3409ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:15:59.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.913+0000 7f341b9c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:15:59.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.913+0000 7f341b9c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:15:59.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.913+0000 7f341b9c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:15:59.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.913+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3414139b40 con 0x7f341406e0e0 2024-09-11T05:15:59.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.913+0000 7f3409ffb640 1 -- 172.21.15.138:0/1735370487 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f34140a3270 con 0x7f341406e0e0 2024-09-11T05:15:59.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f3409ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:15:59.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f3409ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:15:59.918 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:15:59.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 10 librados: watch_flush enter 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 10 librados: watch_flush exit 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 20 client.64407.objecter shutdown clearing up homeless session... 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 10 client.64407.objecter successfully canceled tick 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 >> v2:172.21.15.190:6800/1562782189 conn(0x7f33f803cf20 msgr2=0x7f33f803f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 --2- 172.21.15.138:0/1735370487 >> v2:172.21.15.190:6800/1562782189 conn(0x7f33f803cf20 0x7f33f803f3e0 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 crypto rx=0x7f3414139dc0 tx=0x7f3404008040 comp rx=0 tx=0).stop 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 10 monclient: shutdown 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 msgr2=0x7f34141ce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f34141ce910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f34100af040 tx=0x7f3410018f00 comp rx=0 tx=0).stop 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 shutdown_connections 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 --2- 172.21.15.138:0/1735370487 >> v2:172.21.15.190:6800/1562782189 conn(0x7f33f803cf20 0x7f33f803f3e0 unknown :-1 s=CLOSED pgs=3416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3414136170 0x7f341413dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34140a1920 0x7f341413d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 --2- 172.21.15.138:0/1735370487 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f341406e0e0 0x7f34141ce910 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:15:59.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 >> 172.21.15.138:0/1735370487 conn(0x7f3414063640 msgr2=0x7f34140a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:15:59.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 shutdown_connections 2024-09-11T05:15:59.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 -- 172.21.15.138:0/1735370487 wait complete. 2024-09-11T05:15:59.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:15:59.917+0000 7f341b9c6640 1 librados: shutdown 2024-09-11T05:16:02.938 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient: get_monmap_and_config 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient: build_initial_monmap 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient: monmap: 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:03.070718+0000 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:03.070718+0000 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:03.068 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:03.069 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:03.069 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:03.069 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:03.069 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:03.069 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:03.069 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:03.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 1 Processor -- start 2024-09-11T05:16:03.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 1 -- start start 2024-09-11T05:16:03.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient: init 2024-09-11T05:16:03.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:03.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe54006e0e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 0x7fe54013a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient(hunting): picked mon.noname-b con 0x7fe54009eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe54009ef10 0x7fe54013ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient(hunting): picked mon.noname-c con 0x7fe54009ef10 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54009f540 con 0x7fe54006e0e0 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe54006e4c0 con 0x7fe54009ef10 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe54009f820 con 0x7fe54009eb30 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:03.072 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe545bbd640 10 monclient(hunting): authenticate will time out at 1446.506774s 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 0x7fe54013a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 0x7fe54013a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:37060/0 (socket says 172.21.15.138:37060) 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 1 -- 172.21.15.138:0/4031811443 learned_addr learned my addr 172.21.15.138:0/4031811443 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 10 monclient(hunting): get_auth_request con 0x7fe54009eb30 auth_method 0 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 1 --2- 172.21.15.138:0/4031811443 >> v2:172.21.15.190:3300/0 conn(0x7fe54009ef10 0x7fe54013ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 1 -- 172.21.15.138:0/4031811443 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 msgr2=0x7fe54013a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 1 -- 172.21.15.138:0/4031811443 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 msgr2=0x7fe54013a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 1 --2- 172.21.15.138:0/4031811443 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 0x7fe54013a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53effd640 1 --2- 172.21.15.138:0/4031811443 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 0x7fe54013a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient(hunting): get_auth_request con 0x7fe54009ef10 auth_method 0 2024-09-11T05:16:03.073 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient(hunting): handle_auth_done global_id 74152 payload 274 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 1 -- 172.21.15.138:0/4031811443 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 msgr2=0x7fe54013a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 1 --2- 172.21.15.138:0/4031811443 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 0x7fe54013a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 1 -- 172.21.15.138:0/4031811443 >> v1:172.21.15.190:6789/0 conn(0x7fe54006e0e0 legacy=0x7fe54009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 1 monclient: found mon.noname-c 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:16:03.074 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.065+0000 7fe53ffff640 1 -- 172.21.15.138:0/4031811443 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54009fb00 con 0x7fe54009ef10 2024-09-11T05:16:03.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 -- 172.21.15.138:0/4031811443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009ef10 msgr2=0x7fe54013ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:03.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 --2- 172.21.15.138:0/4031811443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009ef10 0x7fe54013ce80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe538009dc0 tx=0x7fe53801c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fd2602b5a06715e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:03.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 1 -- 172.21.15.138:0/4031811443 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380b2070 con 0x7fe54009ef10 2024-09-11T05:16:03.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:03.076 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: dump: 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:16:03.077 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 1 -- 172.21.15.138:0/4031811443 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5380a5790 con 0x7fe54009ef10 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 1 -- 172.21.15.138:0/4031811443 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380a5b60 con 0x7fe54009ef10 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53e7fc640 10 monclient: dump: 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:03.078 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 5 monclient: authenticate success, global_id 74152 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 4 set_mon_vals no callback set 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient: shutdown 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- 172.21.15.138:0/4031811443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009ef10 msgr2=0x7fe54013ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 --2- 172.21.15.138:0/4031811443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009ef10 0x7fe54013ce80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe538009dc0 tx=0x7fe53801c810 comp rx=0 tx=0).stop 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- 172.21.15.138:0/4031811443 shutdown_connections 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 --2- 172.21.15.138:0/4031811443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009ef10 0x7fe54013ce80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 --2- 172.21.15.138:0/4031811443 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54009eb30 0x7fe54013a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- 172.21.15.138:0/4031811443 >> 172.21.15.138:0/4031811443 conn(0x7fe540063640 msgr2=0x7fe540063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- 172.21.15.138:0/4031811443 shutdown_connections 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- 172.21.15.138:0/4031811443 wait complete. 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient: build_initial_monmap 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient: monmap: 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:03.080 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 librados: starting msgr at 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 librados: starting objecter 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 Processor -- start 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- start start 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 librados: setting wanted keys 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 librados: calling monclient init 2024-09-11T05:16:03.081 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient: init 2024-09-11T05:16:03.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:03.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:16:03.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 0x7fe5401356c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:03.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient(hunting): picked mon.noname-c con 0x7fe54006e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:03.085 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009eb30 0x7fe54011d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:03.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient(hunting): picked mon.noname-b con 0x7fe54009eb30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:03.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe54009ef10 0x7fe54011dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:03.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient(hunting): picked mon.noname-a con 0x7fe54009ef10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:03.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:03.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe54013fdf0 con 0x7fe54009ef10 2024-09-11T05:16:03.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:03.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe5400a1400 con 0x7fe54009eb30 2024-09-11T05:16:03.088 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe54013ff70 con 0x7fe54006e0e0 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe545bbd640 10 monclient(hunting): authenticate will time out at 1446.515075s 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 0x7fe5401356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 0x7fe5401356c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:37076/0 (socket says 172.21.15.138:37076) 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 1 -- 172.21.15.138:0/2899903892 learned_addr learned my addr 172.21.15.138:0/2899903892 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 10 monclient(hunting): get_auth_request con 0x7fe54006e0e0 auth_method 0 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:03.089 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe54009ef10 0x7fe54011dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient(hunting): get_auth_request con 0x7fe54009ef10 auth_method 0 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 1 -- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 msgr2=0x7fe5401356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 1 -- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 msgr2=0x7fe5401356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 0x7fe5401356c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53f7fe640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 0x7fe5401356c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:03.090 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53effd640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009eb30 0x7fe54011d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53effd640 10 monclient(hunting): get_auth_request con 0x7fe54009eb30 auth_method 0 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient(hunting): handle_auth_done global_id 64410 payload 824 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 -- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 msgr2=0x7fe5401356c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 0x7fe5401356c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 -- 172.21.15.138:0/2899903892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009eb30 msgr2=0x7fe54011d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009eb30 0x7fe54011d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 monclient: found mon.noname-a 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:03.091 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.073+0000 7fe53ffff640 1 -- 172.21.15.138:0/2899903892 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54011ed00 con 0x7fe54009ef10 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe53ffff640 1 -- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe54009ef10 msgr2=0x7fe54011dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe53ffff640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe54009ef10 0x7fe54011dab0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe5380af000 tx=0x7fe5380195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 client.?.objecter ms_handle_connect 0x7fe54009ef10 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 1 -- 172.21.15.138:0/2899903892 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380bb020 con 0x7fe54009ef10 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: dump: 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:03.092 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: _finish_auth 0 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: _check_auth_tickets 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 1 -- 172.21.15.138:0/2899903892 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5380ad650 con 0x7fe54009ef10 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 5 monclient: authenticate success, global_id 64410 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 10 monclient: _renew_subs 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:03.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54009e110 con 0x7fe54009ef10 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 1 -- 172.21.15.138:0/2899903892 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380b2040 con 0x7fe54009ef10 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 monclient: dump: 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:03.094 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 10 client.64410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 10 monclient: _renew_subs 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe540121060 con 0x7fe54009ef10 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 1 librados: init done 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe53dffb640 4 set_mon_vals no callback set 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe54009e420 con 0x7fe54009ef10 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 1 -- 172.21.15.138:0/2899903892 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fe5380ada80 con 0x7fe54009ef10 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 1 --2- 172.21.15.138:0/2899903892 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe52c03cea0 0x7fe52c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 1 -- 172.21.15.138:0/2899903892 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(362..362 src has 1..362) ==== 5390+0+0 (secure 0 0 0) 0x7fe5380ea390 con 0x7fe54009ef10 2024-09-11T05:16:03.095 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 10 client.64410.objecter ms_dispatch 0x7fe540000ba0 osd_map(362..362 src has 1..362) 2024-09-11T05:16:03.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 3 client.64410.objecter handle_osd_map got epochs [362,362] > 0 2024-09-11T05:16:03.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 3 client.64410.objecter handle_osd_map decoding full epoch 362 2024-09-11T05:16:03.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.077+0000 7fe527fff640 20 client.64410.objecter dump_active .. 0 homeless 2024-09-11T05:16:03.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.081+0000 7fe53f7fe640 1 --2- 172.21.15.138:0/2899903892 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe52c03cea0 0x7fe52c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:03.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.081+0000 7fe53f7fe640 10 monclient: get_auth_request con 0x7fe52c03cea0 auth_method 0 2024-09-11T05:16:03.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.081+0000 7fe53f7fe640 1 --2- 172.21.15.138:0/2899903892 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe52c03cea0 0x7fe52c03f360 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7fe530006fd0 tx=0x7fe530008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:03.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.081+0000 7fe527fff640 10 client.64410.objecter ms_handle_connect 0x7fe52c03cea0 2024-09-11T05:16:03.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.085+0000 7fe527fff640 1 -- 172.21.15.138:0/2899903892 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe5380c5070 con 0x7fe54009ef10 2024-09-11T05:16:03.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.085+0000 7fe527fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:03.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.085+0000 7fe527fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:03.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.317+0000 7fe545bbd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:16:03.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.317+0000 7fe545bbd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:16:03.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.317+0000 7fe545bbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:03.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.317+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe54009dea0 con 0x7fe54009ef10 2024-09-11T05:16:03.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe527fff640 1 -- 172.21.15.138:0/2899903892 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fe538016520 con 0x7fe54009ef10 2024-09-11T05:16:03.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe527fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:16:03.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe527fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:03.323 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:16:03.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 10 librados: watch_flush enter 2024-09-11T05:16:03.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 10 librados: watch_flush exit 2024-09-11T05:16:03.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 20 client.64410.objecter shutdown clearing up homeless session... 2024-09-11T05:16:03.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 10 client.64410.objecter successfully canceled tick 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe52c03cea0 msgr2=0x7fe52c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 --2- 172.21.15.138:0/2899903892 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe52c03cea0 0x7fe52c03f360 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7fe530006fd0 tx=0x7fe530008040 comp rx=0 tx=0).stop 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 10 monclient: shutdown 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe54009ef10 msgr2=0x7fe54011dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe54009ef10 0x7fe54011dab0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe5380af000 tx=0x7fe5380195b0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 shutdown_connections 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 --2- 172.21.15.138:0/2899903892 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe52c03cea0 0x7fe52c03f360 unknown :-1 s=CLOSED pgs=3417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe54009ef10 0x7fe54011dab0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe54009eb30 0x7fe54011d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 --2- 172.21.15.138:0/2899903892 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe54006e0e0 0x7fe5401356c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:03.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 >> 172.21.15.138:0/2899903892 conn(0x7fe540063640 msgr2=0x7fe5400a29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:03.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 shutdown_connections 2024-09-11T05:16:03.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 -- 172.21.15.138:0/2899903892 wait complete. 2024-09-11T05:16:03.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:03.321+0000 7fe545bbd640 1 librados: shutdown 2024-09-11T05:16:06.342 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.465+0000 7fd0e98c3640 10 monclient: get_monmap_and_config 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.465+0000 7fd0e98c3640 10 monclient: build_initial_monmap 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.465+0000 7fd0e98c3640 10 monclient: monmap: 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:06.472123+0000 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:06.472123+0000 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:06.469 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.470 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:06.470 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:06.470 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:06.470 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:06.470 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:06.470 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.465+0000 7fd0e98c3640 1 Processor -- start 2024-09-11T05:16:06.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.465+0000 7fd0e98c3640 1 -- start start 2024-09-11T05:16:06.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.465+0000 7fd0e98c3640 10 monclient: init 2024-09-11T05:16:06.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:06.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-11T05:16:06.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0e40a4a30 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd0e406fd10 0x7fd0e413dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e406fd10 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 0x7fd0e4140150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e40a5520 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e406f180 con 0x7fd0e40a4a30 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd0e406f500 con 0x7fd0e406fd10 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e406ef80 con 0x7fd0e40a5520 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e98c3640 10 monclient(hunting): authenticate will time out at 1449.908679s 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e2ffd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fd0e40a4a30 0x7fd0e40a4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:60416/0 (socket says 172.21.15.138:60416) 2024-09-11T05:16:06.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e2ffd640 1 -- 172.21.15.138:0/377109858 learned_addr learned my addr 172.21.15.138:0/377109858 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 1 --2- 172.21.15.138:0/377109858 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 0x7fd0e4140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 10 monclient(hunting): get_auth_request con 0x7fd0e40a5520 auth_method 0 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 1 -- 172.21.15.138:0/377109858 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 msgr2=0x7fd0e4140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 1 -- 172.21.15.138:0/377109858 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 msgr2=0x7fd0e4140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 1 --2- 172.21.15.138:0/377109858 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 0x7fd0e4140150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e37fe640 1 --2- 172.21.15.138:0/377109858 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 0x7fd0e4140150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 1 --2- 172.21.15.138:0/377109858 >> v2:172.21.15.190:3300/0 conn(0x7fd0e406fd10 0x7fd0e413dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient(hunting): get_auth_request con 0x7fd0e406fd10 auth_method 0 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient(hunting): handle_auth_done global_id 74158 payload 274 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 1 -- 172.21.15.138:0/377109858 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 msgr2=0x7fd0e4140150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 1 --2- 172.21.15.138:0/377109858 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 0x7fd0e4140150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 1 -- 172.21.15.138:0/377109858 >> v1:172.21.15.138:6789/0 conn(0x7fd0e40a4a30 legacy=0x7fd0e40a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient: _finish_hunting 0 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 1 monclient: found mon.noname-c 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:16:06.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.469+0000 7fd0e27fc640 1 -- 172.21.15.138:0/377109858 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e40704a0 con 0x7fd0e406fd10 2024-09-11T05:16:06.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e27fc640 1 -- 172.21.15.138:0/377109858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 msgr2=0x7fd0e413dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:06.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e27fc640 1 --2- 172.21.15.138:0/377109858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 0x7fd0e413dc00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0097c0 tx=0x7fd0cc01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78f556e502641183 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:06.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 1 -- 172.21.15.138:0/377109858 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc0b0070 con 0x7fd0e406fd10 2024-09-11T05:16:06.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: dump: 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: _finish_auth 0 2024-09-11T05:16:06.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 1 -- 172.21.15.138:0/377109858 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0cc0ab600 con 0x7fd0e406fd10 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 1 -- 172.21.15.138:0/377109858 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc0ab9d0 con 0x7fd0e406fd10 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e1ffb640 10 monclient: dump: 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.478 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.479 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.479 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.479 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 5 monclient: authenticate success, global_id 74158 2024-09-11T05:16:06.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:06.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 4 set_mon_vals no callback set 2024-09-11T05:16:06.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 10 monclient: shutdown 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 1 -- 172.21.15.138:0/377109858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 msgr2=0x7fd0e413dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/377109858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 0x7fd0e413dc00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0097c0 tx=0x7fd0cc01c360 comp rx=0 tx=0).stop 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 1 -- 172.21.15.138:0/377109858 shutdown_connections 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/377109858 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a5520 0x7fd0e4140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/377109858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 0x7fd0e413dc00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 1 -- 172.21.15.138:0/377109858 >> 172.21.15.138:0/377109858 conn(0x7fd0e40a96e0 msgr2=0x7fd0e40a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 1 -- 172.21.15.138:0/377109858 shutdown_connections 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.473+0000 7fd0e98c3640 1 -- 172.21.15.138:0/377109858 wait complete. 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient: build_initial_monmap 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient: monmap: 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:06.483 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 librados: starting msgr at 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 librados: starting objecter 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 Processor -- start 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 -- start start 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 librados: setting wanted keys 2024-09-11T05:16:06.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 librados: calling monclient init 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient: init 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 0x7fd0e4126840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e406fd10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 0x7fd0e4128980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e40a4a30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0e40a5520 0x7fd0e413f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e40a5520 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e4143240 con 0x7fd0e40a5520 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e41430c0 con 0x7fd0e406fd10 2024-09-11T05:16:06.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e41433c0 con 0x7fd0e40a4a30 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e98c3640 10 monclient(hunting): authenticate will time out at 1449.916969s 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0e40a5520 0x7fd0e413f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 0x7fd0e4128980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0e40a5520 0x7fd0e413f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37960/0 (socket says 172.21.15.138:37960) 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 -- 172.21.15.138:0/2675340679 learned_addr learned my addr 172.21.15.138:0/2675340679 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient(hunting): get_auth_request con 0x7fd0e40a5520 auth_method 0 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 10 monclient(hunting): get_auth_request con 0x7fd0e40a4a30 auth_method 0 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 1 -- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 msgr2=0x7fd0e4128980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 1 -- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 msgr2=0x7fd0e4128980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 0x7fd0e4128980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e27fc640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 0x7fd0e4128980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e2ffd640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 0x7fd0e4126840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient(hunting): handle_auth_done global_id 64413 payload 824 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 -- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 msgr2=0x7fd0e4128980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 0x7fd0e4128980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 -- 172.21.15.138:0/2675340679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 msgr2=0x7fd0e4126840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 0x7fd0e4126840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 monclient: found mon.noname-a 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 -- 172.21.15.138:0/2675340679 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e4127e20 con 0x7fd0e40a5520 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e2ffd640 10 monclient: get_auth_request con 0x7fd0e406fd10 auth_method 0 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e2ffd640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 0x7fd0e4126840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 -- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0e40a5520 msgr2=0x7fd0e413f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0e37fe640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0e40a5520 0x7fd0e413f7f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc007e60 tx=0x7fd0dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0dbfff640 10 client.?.objecter ms_handle_connect 0x7fd0e40a5520 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.477+0000 7fd0dbfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:06.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 1 -- 172.21.15.138:0/2675340679 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc013790 con 0x7fd0e40a5520 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: dump: 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: _finish_auth 0 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: _check_auth_tickets 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 1 -- 172.21.15.138:0/2675340679 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0dc019070 con 0x7fd0e40a5520 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 1 -- 172.21.15.138:0/2675340679 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc013b70 con 0x7fd0e40a5520 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 monclient: dump: 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.490 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 5 monclient: authenticate success, global_id 64413 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 10 monclient: _renew_subs 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e40a0210 con 0x7fd0e40a5520 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e17fa640 4 set_mon_vals no callback set 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 10 client.64413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 10 monclient: _renew_subs 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e4127750 con 0x7fd0e40a5520 2024-09-11T05:16:06.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 1 librados: init done 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e40a4e10 con 0x7fd0e40a5520 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 1 -- 172.21.15.138:0/2675340679 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fd0dc014420 con 0x7fd0e40a5520 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 1 --2- 172.21.15.138:0/2675340679 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd0c403cef0 0x7fd0c403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 1 -- 172.21.15.138:0/2675340679 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (secure 0 0 0) 0x7fd0dc050100 con 0x7fd0e40a5520 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 10 client.64413.objecter ms_dispatch 0x7fd0e4000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 3 client.64413.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 3 client.64413.objecter handle_osd_map decoding full epoch 365 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0dbfff640 20 client.64413.objecter dump_active .. 0 homeless 2024-09-11T05:16:06.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e2ffd640 1 --2- 172.21.15.138:0/2675340679 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd0c403cef0 0x7fd0c403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.481+0000 7fd0e2ffd640 10 monclient: get_auth_request con 0x7fd0c403cef0 auth_method 0 2024-09-11T05:16:06.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.485+0000 7fd0e2ffd640 1 --2- 172.21.15.138:0/2675340679 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd0c403cef0 0x7fd0c403f3b0 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7fd0e41297d0 tx=0x7fd0d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:06.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.485+0000 7fd0dbfff640 10 client.64413.objecter ms_handle_connect 0x7fd0c403cef0 2024-09-11T05:16:06.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.485+0000 7fd0dbfff640 1 -- 172.21.15.138:0/2675340679 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fd0dc013d90 con 0x7fd0e40a5520 2024-09-11T05:16:06.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.485+0000 7fd0dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:06.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.485+0000 7fd0dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:06.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.721+0000 7fd0e98c3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:16:06.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.721+0000 7fd0e98c3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:16:06.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.721+0000 7fd0e98c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:06.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.721+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd0e406cc20 con 0x7fd0e40a5520 2024-09-11T05:16:06.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.721+0000 7fd0dbfff640 1 -- 172.21.15.138:0/2675340679 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fd0dc014740 con 0x7fd0e40a5520 2024-09-11T05:16:06.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.721+0000 7fd0dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:16:06.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.721+0000 7fd0dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:06.726 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:16:06.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 10 librados: watch_flush enter 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 10 librados: watch_flush exit 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 20 client.64413.objecter shutdown clearing up homeless session... 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 10 client.64413.objecter successfully canceled tick 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd0c403cef0 msgr2=0x7fd0c403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/2675340679 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd0c403cef0 0x7fd0c403f3b0 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7fd0e41297d0 tx=0x7fd0d4009040 comp rx=0 tx=0).stop 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 10 monclient: shutdown 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0e40a5520 msgr2=0x7fd0e413f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0e40a5520 0x7fd0e413f7f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc007e60 tx=0x7fd0dc00e4b0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 shutdown_connections 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/2675340679 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd0c403cef0 0x7fd0c403f3b0 unknown :-1 s=CLOSED pgs=3418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd0e40a5520 0x7fd0e413f7f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.729 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd0e40a4a30 0x7fd0e4128980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 --2- 172.21.15.138:0/2675340679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0e406fd10 0x7fd0e4126840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 >> 172.21.15.138:0/2675340679 conn(0x7fd0e40a96e0 msgr2=0x7fd0e40a7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:06.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 shutdown_connections 2024-09-11T05:16:06.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 -- 172.21.15.138:0/2675340679 wait complete. 2024-09-11T05:16:06.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.725+0000 7fd0e98c3640 1 librados: shutdown 2024-09-11T05:16:06.745 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:16:06.745 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:16:06.745 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:16:06.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.865+0000 7f35cf78c640 10 monclient: get_monmap_and_config 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.865+0000 7f35cf78c640 10 monclient: build_initial_monmap 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.865+0000 7f35cf78c640 10 monclient: monmap: 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:06.872898+0000 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:06.872898+0000 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:06.870 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:06.871 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:06.871 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 1 Processor -- start 2024-09-11T05:16:06.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 1 -- start start 2024-09-11T05:16:06.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient: init 2024-09-11T05:16:06.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:06.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-11T05:16:06.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f35c8070010 0x7f35c8135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient(hunting): picked mon.noname-c con 0x7f35c8070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:06.873 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35c809e6e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 0x7f35c81389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient(hunting): picked mon.noname-b con 0x7f35c809eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c80a8990 con 0x7f35c809e6e0 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f35c80703f0 con 0x7f35c8070010 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f35c809cbf0 con 0x7f35c809eb10 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cf78c640 10 monclient(hunting): authenticate will time out at 1450.309435s 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 0x7f35c81389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 0x7f35c81389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:37102/0 (socket says 172.21.15.138:37102) 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 1 -- 172.21.15.138:0/3316868544 learned_addr learned my addr 172.21.15.138:0/3316868544 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 10 monclient(hunting): get_auth_request con 0x7f35c809eb10 auth_method 0 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.874 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 1 --2- 172.21.15.138:0/3316868544 >> v2:172.21.15.190:3300/0 conn(0x7f35c8070010 0x7f35c8135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 1 -- 172.21.15.138:0/3316868544 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 msgr2=0x7f35c81389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 1 -- 172.21.15.138:0/3316868544 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 msgr2=0x7f35c81389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 1 --2- 172.21.15.138:0/3316868544 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 0x7f35c81389d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cdd29640 1 --2- 172.21.15.138:0/3316868544 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 0x7f35c81389d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient(hunting): get_auth_request con 0x7f35c8070010 auth_method 0 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient(hunting): handle_auth_done global_id 74161 payload 274 2024-09-11T05:16:06.875 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 1 -- 172.21.15.138:0/3316868544 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 msgr2=0x7f35c81389d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 1 --2- 172.21.15.138:0/3316868544 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 0x7f35c81389d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 1 -- 172.21.15.138:0/3316868544 >> v1:172.21.15.190:6789/0 conn(0x7f35c809e6e0 legacy=0x7f35c81382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient: _finish_hunting 0 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 1 monclient: found mon.noname-c 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.869+0000 7f35cd528640 1 -- 172.21.15.138:0/3316868544 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c809ced0 con 0x7f35c8070010 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cd528640 1 -- 172.21.15.138:0/3316868544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c8070010 msgr2=0x7f35c8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:06.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cd528640 1 --2- 172.21.15.138:0/3316868544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c8070010 0x7f35c8135450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f35c40097c0 tx=0x7f35c401c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a335759188acb5f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 1 -- 172.21.15.138:0/3316868544 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c40b0070 con 0x7f35c8070010 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: dump: 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.877 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: _finish_auth 0 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: _check_auth_tickets 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 1 -- 172.21.15.138:0/3316868544 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c40ab600 con 0x7f35c8070010 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 1 -- 172.21.15.138:0/3316868544 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c40ab9d0 con 0x7f35c8070010 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 5 monclient: authenticate success, global_id 74161 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 4 set_mon_vals no callback set 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.878 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35bffff640 10 monclient: dump: 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.879 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 10 monclient: shutdown 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 1 -- 172.21.15.138:0/3316868544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c8070010 msgr2=0x7f35c8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 1 --2- 172.21.15.138:0/3316868544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c8070010 0x7f35c8135450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f35c40097c0 tx=0x7f35c401c330 comp rx=0 tx=0).stop 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 1 -- 172.21.15.138:0/3316868544 shutdown_connections 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 1 --2- 172.21.15.138:0/3316868544 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809eb10 0x7f35c81389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 1 --2- 172.21.15.138:0/3316868544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c8070010 0x7f35c8135450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 1 -- 172.21.15.138:0/3316868544 >> 172.21.15.138:0/3316868544 conn(0x7f35c80ab830 msgr2=0x7f35c80a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 1 -- 172.21.15.138:0/3316868544 shutdown_connections 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.873+0000 7f35cf78c640 1 -- 172.21.15.138:0/3316868544 wait complete. 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient: build_initial_monmap 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient: monmap: 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:06.885 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 librados: starting msgr at 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 librados: starting objecter 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 Processor -- start 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 -- start start 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 librados: setting wanted keys 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 librados: calling monclient init 2024-09-11T05:16:06.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient: init 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 0x7f35c81d0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient(hunting): picked mon.noname-a con 0x7f35c8070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809e6e0 0x7f35c81d0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient(hunting): picked mon.noname-c con 0x7f35c809e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c809eb10 0x7f35c81d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient(hunting): picked mon.noname-b con 0x7f35c809eb10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35c8142e80 con 0x7f35c8070010 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f35c8139120 con 0x7f35c809eb10 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f35c8143000 con 0x7f35c809e6e0 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cf78c640 10 monclient(hunting): authenticate will time out at 1450.317617s 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 0x7f35c81d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 0x7f35c81d0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37970/0 (socket says 172.21.15.138:37970) 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 -- 172.21.15.138:0/3293404360 learned_addr learned my addr 172.21.15.138:0/3293404360 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient(hunting): get_auth_request con 0x7f35c8070010 auth_method 0 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809e6e0 0x7f35c81d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 10 monclient(hunting): get_auth_request con 0x7f35c809e6e0 auth_method 0 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cdd29640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c809eb10 0x7f35c81d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809e6e0 msgr2=0x7f35c81d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cdd29640 10 monclient(hunting): get_auth_request con 0x7f35c809eb10 auth_method 0 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809e6e0 msgr2=0x7f35c81d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cdd29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cdd29640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cdd29640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809e6e0 0x7f35c81d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35ccd27640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809e6e0 0x7f35c81d0940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:06.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cdd29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cdd29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cdd29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient(hunting): handle_auth_done global_id 64416 payload 824 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809e6e0 msgr2=0x7f35c81d0940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f35c809e6e0 0x7f35c81d0940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c809eb10 msgr2=0x7f35c81d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f35c809eb10 0x7f35c81d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient: _finish_hunting 0 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 monclient: found mon.noname-a 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.877+0000 7f35cd528640 1 -- 172.21.15.138:0/3293404360 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c81327d0 con 0x7f35c8070010 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cd528640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 msgr2=0x7f35c81d0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:06.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cd528640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 0x7f35c81d0400 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35c40ae040 tx=0x7f35c401ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 client.?.objecter ms_handle_connect 0x7f35c8070010 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c40b0050 con 0x7f35c8070010 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: dump: 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.891 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: _finish_auth 0 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: _check_auth_tickets 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 5 monclient: authenticate success, global_id 64416 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c40bc070 con 0x7f35c8070010 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 10 monclient: _renew_subs 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c81337b0 con 0x7f35c8070010 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c40b5c80 con 0x7f35c8070010 2024-09-11T05:16:06.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 10 monclient: dump: 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bf7fe640 4 set_mon_vals no callback set 2024-09-11T05:16:06.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 10 client.64416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 10 monclient: _renew_subs 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c8130f30 con 0x7f35c8070010 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 1 librados: init done 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812e860 0x7f35c812ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35c809e550 con 0x7f35c812e860 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812e860 0x7f35c812ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 10 monclient: get_auth_request con 0x7f35c812e860 auth_method 0 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:06.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:06.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:06.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cd528640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 0x7f35c81d0400 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35c40ae040 tx=0x7f35c401ab70 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f35c40c1460 mgrmap(e 5) 2024-09-11T05:16:06.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f35c40c1460 con 0x7f35c8070010 2024-09-11T05:16:06.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 10 monclient(hunting): handle_auth_done global_id 64419 payload 824 2024-09-11T05:16:06.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35ccd27640 1 -- 172.21.15.138:0/3293404360 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35c81331f0 con 0x7f35c812e860 2024-09-11T05:16:06.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35bdffb640 1 --2- 172.21.15.138:0/3293404360 >> v2:172.21.15.190:6800/1562782189 conn(0x7f35ac03ced0 0x7f35ac03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cdd29640 1 --2- 172.21.15.138:0/3293404360 >> v2:172.21.15.190:6800/1562782189 conn(0x7f35ac03ced0 0x7f35ac03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.881+0000 7f35cdd29640 10 monclient: get_auth_request con 0x7f35ac03ced0 auth_method 0 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 10 client.64416.objecter ms_handle_connect 0x7f35c812e860 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 10 client.64416.objecter resend_mon_ops 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35ccd27640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812e860 msgr2=0x7f35c812ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35ccd27640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812e860 0x7f35c812ec40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f35b8007c60 tx=0x7f35b800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b801a070 con 0x7f35c812e860 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 10 monclient: dump: 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.898 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f35b8007450 con 0x7f35c812e860 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812e860 msgr2=0x7f35c812ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812e860 0x7f35c812ec40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f35b8007c60 tx=0x7f35b800e960 comp rx=0 tx=0).stop 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812e860 msgr2=0x7f35c812ec40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812e860 0x7f35c812ec40 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f35b8007c60 tx=0x7f35b800e960 comp rx=0 tx=0).stop 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35cdd29640 1 --2- 172.21.15.138:0/3293404360 >> v2:172.21.15.190:6800/1562782189 conn(0x7f35ac03ced0 0x7f35ac03f390 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7f35c80a3470 tx=0x7f35c0007070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:06.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.885+0000 7f35bdffb640 10 client.64416.objecter ms_handle_connect 0x7f35ac03ced0 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35cf78c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35cf78c640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812f180 0x7f35c8135e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35cf78c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f35c80a4120 con 0x7f35c812f180 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812f180 0x7f35c8135e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 10 monclient: get_auth_request con 0x7f35c812f180 auth_method 0 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 10 monclient(hunting): handle_auth_done global_id 64422 payload 824 2024-09-11T05:16:06.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.889+0000 7f35ccd27640 1 -- 172.21.15.138:0/3293404360 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35c8135450 con 0x7f35c812f180 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 client.64416.objecter ms_handle_connect 0x7f35c812f180 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 client.64416.objecter resend_mon_ops 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35ccd27640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812f180 msgr2=0x7f35c8135e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35ccd27640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812f180 0x7f35c8135e90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f35b8007c60 tx=0x7f35b8003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b801a030 con 0x7f35c812f180 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 monclient: dump: 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:06.901 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f35b801b040 con 0x7f35c812f180 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812f180 msgr2=0x7f35c8135e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812f180 0x7f35c8135e90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f35b8007c60 tx=0x7f35b8003770 comp rx=0 tx=0).stop 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812f180 msgr2=0x7f35c8135e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c812f180 0x7f35c8135e90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.902 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 32, 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 270523, 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:16:06.903 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:06.904 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:06.905 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:16:06.906 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:06.907 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:06.908 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:06.909 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:16:06.910 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:06.911 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:16:06.912 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:06.913 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:06.914 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:16:06.915 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:16:06.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 10 librados: watch_flush enter 2024-09-11T05:16:06.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 10 librados: watch_flush exit 2024-09-11T05:16:06.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 20 client.64416.objecter shutdown clearing up homeless session... 2024-09-11T05:16:06.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 10 client.64416.objecter successfully canceled tick 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 >> v2:172.21.15.190:6800/1562782189 conn(0x7f35ac03ced0 msgr2=0x7f35ac03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 --2- 172.21.15.138:0/3293404360 >> v2:172.21.15.190:6800/1562782189 conn(0x7f35ac03ced0 0x7f35ac03f390 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7f35c80a3470 tx=0x7f35c0007070 comp rx=0 tx=0).stop 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 10 monclient: shutdown 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 msgr2=0x7f35c81d0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 0x7f35c81d0400 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35c40ae040 tx=0x7f35c401ab70 comp rx=0 tx=0).stop 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cdd29640 1 -- 172.21.15.138:0/3293404360 reap_dead start 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 1 -- 172.21.15.138:0/3293404360 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (unknown 0 0 0) 0x7f35c40ea2b0 con 0x7f35c8070010 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 client.64416.objecter ms_dispatch 0x7f35c8000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35bdffb640 10 librados: disconnected, discarding osd_map(365..365 src has 1..365) 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 shutdown_connections 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 --2- 172.21.15.138:0/3293404360 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f35c8070010 0x7f35c81d0400 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 >> 172.21.15.138:0/3293404360 conn(0x7f35c80ab830 msgr2=0x7f35c80a10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:06.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 shutdown_connections 2024-09-11T05:16:06.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 -- 172.21.15.138:0/3293404360 wait complete. 2024-09-11T05:16:06.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:06.893+0000 7f35cf78c640 1 librados: shutdown 2024-09-11T05:16:06.918 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:16:06.918 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:16:06.918 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:16:07.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient: get_monmap_and_config 2024-09-11T05:16:07.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient: build_initial_monmap 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient: monmap: 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:07.041912+0000 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:07.041912+0000 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:07.039 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:07.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 1 Processor -- start 2024-09-11T05:16:07.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 1 -- start start 2024-09-11T05:16:07.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient: init 2024-09-11T05:16:07.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:07.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-11T05:16:07.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 0x7f5254138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:07.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient(hunting): picked mon.noname-b con 0x7f5254138690 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:07.042 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5254138fb0 0x7f5254134d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient(hunting): picked mon.noname-c con 0x7f5254138fb0 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52540a34f0 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52540a24a0 con 0x7f52540a34f0 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f52540a3160 con 0x7f5254138fb0 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f52540a2780 con 0x7f5254138690 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5258876640 10 monclient(hunting): authenticate will time out at 1450.477954s 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 0x7f5254138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 0x7f5254138a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:37128/0 (socket says 172.21.15.138:37128) 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 1 -- 172.21.15.138:0/1796316523 learned_addr learned my addr 172.21.15.138:0/1796316523 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 10 monclient(hunting): get_auth_request con 0x7f5254138690 auth_method 0 2024-09-11T05:16:07.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 1 -- 172.21.15.138:0/1796316523 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 msgr2=0x7f5254138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 1 -- 172.21.15.138:0/1796316523 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 msgr2=0x7f5254138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1796316523 >> v2:172.21.15.190:3300/0 conn(0x7f5254138fb0 0x7f5254134d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 1 --2- 172.21.15.138:0/1796316523 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 0x7f5254138a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f525259c640 1 --2- 172.21.15.138:0/1796316523 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 0x7f5254138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient(hunting): get_auth_request con 0x7f5254138fb0 auth_method 0 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:07.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient(hunting): handle_auth_done global_id 74167 payload 274 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 1 -- 172.21.15.138:0/1796316523 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 msgr2=0x7f5254138a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1796316523 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 0x7f5254138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 1 -- 172.21.15.138:0/1796316523 >> v1:172.21.15.190:6789/0 conn(0x7f52540a34f0 legacy=0x7f52541352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 1 monclient: found mon.noname-c 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:16:07.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.037+0000 7f5251d9b640 1 -- 172.21.15.138:0/1796316523 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52540a2a60 con 0x7f5254138fb0 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f5251d9b640 1 -- 172.21.15.138:0/1796316523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138fb0 msgr2=0x7f5254134d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1796316523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138fb0 0x7f5254134d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52440097c0 tx=0x7f524401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f805a48714d67491 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 1 -- 172.21.15.138:0/1796316523 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52440b0070 con 0x7f5254138fb0 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: dump: 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:07.047 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: _finish_auth 0 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: _check_auth_tickets 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 1 -- 172.21.15.138:0/1796316523 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52440ab600 con 0x7f5254138fb0 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:07.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 1 -- 172.21.15.138:0/1796316523 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52440ab9d0 con 0x7f5254138fb0 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f525159a640 10 monclient: dump: 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:07.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f5258876640 5 monclient: authenticate success, global_id 74167 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f5258876640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.041+0000 7f5258876640 4 set_mon_vals no callback set 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient: shutdown 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- 172.21.15.138:0/1796316523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138fb0 msgr2=0x7f5254134d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 --2- 172.21.15.138:0/1796316523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138fb0 0x7f5254134d90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52440097c0 tx=0x7f524401c360 comp rx=0 tx=0).stop 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- 172.21.15.138:0/1796316523 shutdown_connections 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 --2- 172.21.15.138:0/1796316523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138fb0 0x7f5254134d90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 --2- 172.21.15.138:0/1796316523 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138690 0x7f5254138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- 172.21.15.138:0/1796316523 >> 172.21.15.138:0/1796316523 conn(0x7f5254063640 msgr2=0x7f5254063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- 172.21.15.138:0/1796316523 shutdown_connections 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- 172.21.15.138:0/1796316523 wait complete. 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient: build_initial_monmap 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient: monmap: 2024-09-11T05:16:07.050 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 librados: starting msgr at 2024-09-11T05:16:07.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 librados: starting objecter 2024-09-11T05:16:07.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 Processor -- start 2024-09-11T05:16:07.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- start start 2024-09-11T05:16:07.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 librados: setting wanted keys 2024-09-11T05:16:07.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 librados: calling monclient init 2024-09-11T05:16:07.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient: init 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52540a34f0 0x7f525412e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient(hunting): picked mon.noname-a con 0x7f52540a34f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138690 0x7f525412ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient(hunting): picked mon.noname-b con 0x7f5254138690 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 0x7f5254139420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient(hunting): picked mon.noname-c con 0x7f5254138fb0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5254142eb0 con 0x7f52540a34f0 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5254135bf0 con 0x7f5254138690 2024-09-11T05:16:07.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5254143030 con 0x7f5254138fb0 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5258876640 10 monclient(hunting): authenticate will time out at 1450.486375s 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52540a34f0 0x7f525412e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 0x7f5254139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52540a34f0 0x7f525412e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37998/0 (socket says 172.21.15.138:37998) 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 0x7f5254139420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:37142/0 (socket says 172.21.15.138:37142) 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 -- 172.21.15.138:0/1748090925 learned_addr learned my addr 172.21.15.138:0/1748090925 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient(hunting): get_auth_request con 0x7f52540a34f0 auth_method 0 2024-09-11T05:16:07.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 10 monclient(hunting): get_auth_request con 0x7f5254138fb0 auth_method 0 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 msgr2=0x7f5254139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 msgr2=0x7f5254139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 0x7f5254139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5252d9d640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 0x7f5254139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:07.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138690 0x7f525412ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5251d9b640 10 monclient(hunting): get_auth_request con 0x7f5254138690 auth_method 0 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f5251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient(hunting): handle_auth_done global_id 64425 payload 824 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 msgr2=0x7f5254139420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5254138fb0 0x7f5254139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138690 msgr2=0x7f525412ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5254138690 0x7f525412ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient: _finish_hunting 0 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 monclient: found mon.noname-a 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:07.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.045+0000 7f525259c640 1 -- 172.21.15.138:0/1748090925 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525409e110 con 0x7f52540a34f0 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f525259c640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52540a34f0 msgr2=0x7f525412e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f525259c640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52540a34f0 0x7f525412e9d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f524800e810 tx=0x7f524800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 client.?.objecter ms_handle_connect 0x7f52540a34f0 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248017070 con 0x7f52540a34f0 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: dump: 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:07.061 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: _finish_auth 0 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 5 monclient: authenticate success, global_id 64425 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 10 monclient: _renew_subs 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5248004020 con 0x7f52540a34f0 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5254132320 con 0x7f52540a34f0 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:07.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524800f040 con 0x7f52540a34f0 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 monclient: dump: 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5250d99640 4 set_mon_vals no callback set 2024-09-11T05:16:07.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 10 client.64425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 10 monclient: _renew_subs 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525413bf20 con 0x7f52540a34f0 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 1 librados: init done 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f525412f510 con 0x7f525413a720 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f525413a720 auth_method 0 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:07.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5248012050 con 0x7f52540a34f0 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 1 --2- 172.21.15.138:0/1748090925 >> v2:172.21.15.190:6800/1562782189 conn(0x7f523403cea0 0x7f523403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (secure 0 0 0) 0x7f524804f3b0 con 0x7f52540a34f0 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 10 client.64425.objecter ms_dispatch 0x7f5254000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 3 client.64425.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 3 client.64425.objecter handle_osd_map decoding full epoch 365 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5242ffd640 20 client.64425.objecter dump_active .. 0 homeless 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5252d9d640 1 --2- 172.21.15.138:0/1748090925 >> v2:172.21.15.190:6800/1562782189 conn(0x7f523403cea0 0x7f523403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.049+0000 7f5252d9d640 10 monclient: get_auth_request con 0x7f523403cea0 auth_method 0 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.053+0000 7f5242ffd640 10 client.64425.objecter ms_handle_connect 0x7f523403cea0 2024-09-11T05:16:07.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.053+0000 7f5252d9d640 1 --2- 172.21.15.138:0/1748090925 >> v2:172.21.15.190:6800/1562782189 conn(0x7f523403cea0 0x7f523403f360 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7f524c007fd0 tx=0x7f524c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f525413a720 auth_method 0 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:07.256 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.253+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:16:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f525413a720 auth_method 0 2024-09-11T05:16:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:16:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:07.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:07.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:07.657+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:16:08.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:08.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f525413a720 auth_method 0 2024-09-11T05:16:08.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:08.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:08.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:08.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:16:08.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:08.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:08.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:08.461+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:16:10.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.045+0000 7f5241ffb640 10 monclient: tick 2024-09-11T05:16:10.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.045+0000 7f5241ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:16:10.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.045+0000 7f5241ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:10.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.045+0000 7f5241ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:10.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:10.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f525413a720 auth_method 0 2024-09-11T05:16:10.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:10.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:10.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:10.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:16:10.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:10.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:10.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:10.065+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-11T05:16:11.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 10 monclient: tick 2024-09-11T05:16:11.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:16:11.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:11.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 msgr2=0x7f525402d450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525413a720 0x7f525402d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5241ffb640 1 -- 172.21.15.138:0/1748090925 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f525412f510 con 0x7f522c003550 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f522c003550 auth_method 0 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 msgr2=0x7f522c0059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:16:11.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 msgr2=0x7f522c0059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:11.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:11.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.049+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f522c003550 auth_method 0 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 msgr2=0x7f522c0059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 msgr2=0x7f522c0059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:11.255 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.253+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:16:11.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f522c003550 auth_method 0 2024-09-11T05:16:11.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:11.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 10 monclient(hunting): handle_auth_done global_id 84101 payload 824 2024-09-11T05:16:11.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.657+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f522c0060c0 con 0x7f522c003550 2024-09-11T05:16:11.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 msgr2=0x7f522c0059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:11.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 10 client.64425.objecter ms_handle_connect 0x7f522c003550 2024-09-11T05:16:11.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5254064470 tx=0x7f5244019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:11.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 10 client.64425.objecter resend_mon_ops 2024-09-11T05:16:11.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52440c5020 con 0x7f522c003550 2024-09-11T05:16:11.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:11.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:11.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 10 monclient: dump: 2024-09-11T05:16:11.663 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:11.663 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.664 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.664 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.664 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:11.664 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.664 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:11.664 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:11.664 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:11.664 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f52440c1640 con 0x7f522c003550 2024-09-11T05:16:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 msgr2=0x7f522c0059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5254064470 tx=0x7f5244019100 comp rx=0 tx=0).stop 2024-09-11T05:16:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 msgr2=0x7f522c0059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.661+0000 7f5242ffd640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f522c003550 0x7f522c0059e0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5254064470 tx=0x7f5244019100 comp rx=0 tx=0).stop 2024-09-11T05:16:11.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5258876640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:16:11.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5258876640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525402d990 0x7f525402dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5258876640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f525413ba40 con 0x7f525402d990 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525402d990 0x7f525402dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 10 monclient: get_auth_request con 0x7f525402d990 auth_method 0 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 10 monclient(hunting): handle_auth_done global_id 84104 payload 824 2024-09-11T05:16:11.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f525409f9f0 con 0x7f525402d990 2024-09-11T05:16:11.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525402d990 msgr2=0x7f525402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:11.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5242ffd640 10 client.64425.objecter ms_handle_connect 0x7f525402d990 2024-09-11T05:16:11.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5242ffd640 10 client.64425.objecter resend_mon_ops 2024-09-11T05:16:11.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.673+0000 7f5251d9b640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525402d990 0x7f525402dda0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5244000c10 tx=0x7f5244008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52440b0040 con 0x7f525402d990 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 10 monclient: dump: 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.679 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f5244003620 con 0x7f525402d990 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525402d990 msgr2=0x7f525402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525402d990 0x7f525402dda0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5244000c10 tx=0x7f5244008040 comp rx=0 tx=0).stop 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525402d990 msgr2=0x7f525402dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5252d9d640 1 -- 172.21.15.138:0/1748090925 reap_dead start 2024-09-11T05:16:11.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5242ffd640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f525402d990 0x7f525402dda0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 32, 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 15447, 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:16:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 5, 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:16:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:16:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:16:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:16:11.693 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:16:11.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 10 librados: watch_flush enter 2024-09-11T05:16:11.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 10 librados: watch_flush exit 2024-09-11T05:16:11.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 20 client.64425.objecter shutdown clearing up homeless session... 2024-09-11T05:16:11.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 10 client.64425.objecter successfully canceled tick 2024-09-11T05:16:11.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 >> v2:172.21.15.190:6800/1562782189 conn(0x7f523403cea0 msgr2=0x7f523403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 --2- 172.21.15.138:0/1748090925 >> v2:172.21.15.190:6800/1562782189 conn(0x7f523403cea0 0x7f523403f360 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7f524c007fd0 tx=0x7f524c009040 comp rx=0 tx=0).stop 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 10 monclient: shutdown 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52540a34f0 msgr2=0x7f525412e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52540a34f0 0x7f525412e9d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f524800e810 tx=0x7f524800ecd0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 shutdown_connections 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 --2- 172.21.15.138:0/1748090925 >> v2:172.21.15.190:6800/1562782189 conn(0x7f523403cea0 0x7f523403f360 unknown :-1 s=CLOSED pgs=3420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 --2- 172.21.15.138:0/1748090925 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52540a34f0 0x7f525412e9d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 >> 172.21.15.138:0/1748090925 conn(0x7f5254063640 msgr2=0x7f5254134d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 shutdown_connections 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 -- 172.21.15.138:0/1748090925 wait complete. 2024-09-11T05:16:11.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.677+0000 7f5258876640 1 librados: shutdown 2024-09-11T05:16:11.701 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:16:11.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:16:11.702 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:16:11.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient: get_monmap_and_config 2024-09-11T05:16:11.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient: build_initial_monmap 2024-09-11T05:16:11.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient: monmap: 2024-09-11T05:16:11.826 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:11.826 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.826 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:11.829527+0000 2024-09-11T05:16:11.826 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:11.829527+0000 2024-09-11T05:16:11.827 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:11.827 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.827 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:11.827 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:11.827 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:11.827 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:11.827 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:11.827 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 1 Processor -- start 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 1 -- start start 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient: init 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2090070010 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f209009e6e0 0x7f209013d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient(hunting): picked mon.noname-a con 0x7f209009e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f209009eb10 0x7f209013fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient(hunting): picked mon.noname-b con 0x7f209009eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2090135b60 con 0x7f2090070010 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f20900703f0 con 0x7f209009e6e0 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f20900a8990 con 0x7f209009eb10 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f20964bf640 10 monclient(hunting): authenticate will time out at 1455.265741s 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f209009e6e0 0x7f209013d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208ffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f2090070010 0x7f2090135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:48890/0 (socket says 172.21.15.138:48890) 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208ffff640 1 -- 172.21.15.138:0/318245260 learned_addr learned my addr 172.21.15.138:0/318245260 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient(hunting): get_auth_request con 0x7f209009e6e0 auth_method 0 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 1 --2- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f209009eb10 0x7f209013fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 10 monclient(hunting): get_auth_request con 0x7f209009eb10 auth_method 0 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient(hunting): handle_auth_done global_id 64428 payload 274 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 1 -- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f209009eb10 msgr2=0x7f209013fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 1 --2- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f209009eb10 0x7f209013fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 1 -- 172.21.15.138:0/318245260 >> v1:172.21.15.138:6789/0 conn(0x7f2090070010 legacy=0x7f2090135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:16:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 1 monclient: found mon.noname-a 2024-09-11T05:16:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f208f7fe640 1 -- 172.21.15.138:0/318245260 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209009cbf0 con 0x7f209009e6e0 2024-09-11T05:16:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.825+0000 7f2094a5c640 1 --2- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f209009eb10 0x7f209013fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:16:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208f7fe640 1 -- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009e6e0 msgr2=0x7f209013d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208f7fe640 1 --2- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009e6e0 0x7f209013d640 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f20780097c0 tx=0x7f207801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a6100b2839adc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 1 -- 172.21.15.138:0/318245260 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780b0070 con 0x7f209009e6e0 2024-09-11T05:16:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: dump: 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: _finish_auth 0 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: _check_auth_tickets 2024-09-11T05:16:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 5 monclient: authenticate success, global_id 64428 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 1 -- 172.21.15.138:0/318245260 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20780ab600 con 0x7f209009e6e0 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 4 set_mon_vals no callback set 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 1 -- 172.21.15.138:0/318245260 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780ab9d0 con 0x7f209009e6e0 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f208effd640 10 monclient: dump: 2024-09-11T05:16:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 10 monclient: shutdown 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 1 -- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009e6e0 msgr2=0x7f209013d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 1 --2- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009e6e0 0x7f209013d640 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f20780097c0 tx=0x7f207801c330 comp rx=0 tx=0).stop 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 1 -- 172.21.15.138:0/318245260 shutdown_connections 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 1 --2- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f209009eb10 0x7f209013fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 1 --2- 172.21.15.138:0/318245260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009e6e0 0x7f209013d640 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 1 -- 172.21.15.138:0/318245260 >> 172.21.15.138:0/318245260 conn(0x7f20900ab830 msgr2=0x7f20900a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 1 -- 172.21.15.138:0/318245260 shutdown_connections 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 1 -- 172.21.15.138:0/318245260 wait complete. 2024-09-11T05:16:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 10 monclient: build_initial_monmap 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.829+0000 7f20964bf640 10 monclient: monmap: 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:11.842 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 librados: starting msgr at 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 librados: starting objecter 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 Processor -- start 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 -- start start 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 librados: setting wanted keys 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 librados: calling monclient init 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient: init 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090070010 0x7f20900a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient(hunting): picked mon.noname-b con 0x7f2090070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009eb10 0x7f20900a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient(hunting): picked mon.noname-a con 0x7f209009eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f20900a4170 0x7f20901d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient(hunting): picked mon.noname-c con 0x7f20900a4170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2090142c80 con 0x7f209009eb10 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2090142b00 con 0x7f2090070010 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2090142e00 con 0x7f20900a4170 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient(hunting): authenticate will time out at 1455.272690s 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009eb10 0x7f20900a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009eb10 0x7f20900a3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:52004/0 (socket says 172.21.15.138:52004) 2024-09-11T05:16:11.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f2094a5c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f20900a4170 0x7f20901d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 -- 172.21.15.138:0/3668184414 learned_addr learned my addr 172.21.15.138:0/3668184414 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient(hunting): get_auth_request con 0x7f209009eb10 auth_method 0 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f2094a5c640 10 monclient(hunting): get_auth_request con 0x7f20900a4170 auth_method 0 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f2094a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f2094a5c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f2094a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f2094a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f2094a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f2094a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient(hunting): handle_auth_done global_id 64431 payload 824 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208ffff640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090070010 0x7f20900a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f20900a4170 msgr2=0x7f20901d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f20900a4170 0x7f20901d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090070010 msgr2=0x7f20900a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090070010 0x7f20900a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 monclient: found mon.noname-a 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 -- 172.21.15.138:0/3668184414 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209013ae60 con 0x7f209009eb10 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208ffff640 10 monclient: get_auth_request con 0x7f2090070010 auth_method 0 2024-09-11T05:16:11.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208ffff640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090070010 0x7f20900a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:16:11.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009eb10 msgr2=0x7f20900a3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:11.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208f7fe640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009eb10 0x7f20900a3c30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f20780ae040 tx=0x7f207801ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:11.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 client.?.objecter ms_handle_connect 0x7f209009eb10 2024-09-11T05:16:11.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:11.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780bc070 con 0x7f209009eb10 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: dump: 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: _finish_auth 0 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: _check_auth_tickets 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20780b1040 con 0x7f209009eb10 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 5 monclient: authenticate success, global_id 64431 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient: _renew_subs 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2090133640 con 0x7f209009eb10 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780b0050 con 0x7f209009eb10 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208cff9640 10 monclient: dump: 2024-09-11T05:16:11.852 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 client.64431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient: _renew_subs 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f208e7fc640 4 set_mon_vals no callback set 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20900aaaa0 con 0x7f209009eb10 2024-09-11T05:16:11.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.833+0000 7f20964bf640 1 librados: init done 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f20964bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f20964bf640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20901d08f0 0x7f20901d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f20964bf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2090135450 con 0x7f20901d08f0 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20901d08f0 0x7f20901d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 10 monclient: get_auth_request con 0x7f20901d08f0 auth_method 0 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f2078004230 con 0x7f209009eb10 2024-09-11T05:16:11.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 10 monclient(hunting): handle_auth_done global_id 74173 payload 824 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208ffff640 1 -- 172.21.15.138:0/3668184414 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2090139c10 con 0x7f20901d08f0 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208cff9640 1 --2- 172.21.15.138:0/3668184414 >> v2:172.21.15.190:6800/1562782189 conn(0x7f207403cea0 0x7f207403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (secure 0 0 0) 0x7f20780e9740 con 0x7f209009eb10 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208cff9640 10 client.64431.objecter ms_dispatch 0x7f2090000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208cff9640 3 client.64431.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208cff9640 3 client.64431.objecter handle_osd_map decoding full epoch 365 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f208cff9640 20 client.64431.objecter dump_active .. 0 homeless 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f2094a5c640 1 --2- 172.21.15.138:0/3668184414 >> v2:172.21.15.190:6800/1562782189 conn(0x7f207403cea0 0x7f207403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.837+0000 7f2094a5c640 10 monclient: get_auth_request con 0x7f207403cea0 auth_method 0 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 10 client.64431.objecter ms_handle_connect 0x7f20901d08f0 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 10 client.64431.objecter resend_mon_ops 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208ffff640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20901d08f0 msgr2=0x7f20901d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208ffff640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20901d08f0 0x7f20901d0cd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f208800dbb0 tx=0x7f208800b4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:11.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2088012610 con 0x7f20901d08f0 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 10 monclient: dump: 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 10 client.64431.objecter ms_handle_connect 0x7f207403cea0 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f2094a5c640 1 --2- 172.21.15.138:0/3668184414 >> v2:172.21.15.190:6800/1562782189 conn(0x7f207403cea0 0x7f207403f360 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f2080009b80 tx=0x7f2080009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2088017070 con 0x7f20901d08f0 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20901d08f0 msgr2=0x7f20901d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20901d08f0 0x7f20901d0cd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f208800dbb0 tx=0x7f208800b4a0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20901d08f0 msgr2=0x7f20901d0cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.841+0000 7f208cff9640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20901d08f0 0x7f20901d0cd0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f208800dbb0 tx=0x7f208800b4a0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f20964bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f20964bf640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090135e90 0x7f20901362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f20964bf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f20901d1210 con 0x7f2090135e90 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f208ffff640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090135e90 0x7f20901362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f208ffff640 10 monclient: get_auth_request con 0x7f2090135e90 auth_method 0 2024-09-11T05:16:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f208ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f208ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.845+0000 7f208ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208ffff640 10 monclient(hunting): handle_auth_done global_id 74176 payload 824 2024-09-11T05:16:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208ffff640 1 -- 172.21.15.138:0/3668184414 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2090069fa0 con 0x7f2090135e90 2024-09-11T05:16:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208ffff640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090135e90 msgr2=0x7f20901362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208ffff640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090135e90 0x7f20901362a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f208800dbb0 tx=0x7f2088004cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 10 client.64431.objecter ms_handle_connect 0x7f2090135e90 2024-09-11T05:16:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 10 client.64431.objecter resend_mon_ops 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2088012d00 con 0x7f2090135e90 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 10 monclient: dump: 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:11.860 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f2088012d00 con 0x7f2090135e90 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090135e90 msgr2=0x7f20901362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090135e90 0x7f20901362a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f208800dbb0 tx=0x7f2088004cd0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090135e90 msgr2=0x7f20901362a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.849+0000 7f208cff9640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2090135e90 0x7f20901362a0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f208800dbb0 tx=0x7f2088004cd0 comp rx=0 tx=0).stop 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:11.861 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 32, 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 139406, 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 5, 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:16:11.862 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:16:11.863 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.864 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:16:11.865 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:11.866 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.868 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:16:11.869 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:11.870 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:16:11.871 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.872 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:11.873 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:16:11.874 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:16:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 10 librados: watch_flush enter 2024-09-11T05:16:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 10 librados: watch_flush exit 2024-09-11T05:16:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 20 client.64431.objecter shutdown clearing up homeless session... 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 10 client.64431.objecter successfully canceled tick 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 >> v2:172.21.15.190:6800/1562782189 conn(0x7f207403cea0 msgr2=0x7f207403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 --2- 172.21.15.138:0/3668184414 >> v2:172.21.15.190:6800/1562782189 conn(0x7f207403cea0 0x7f207403f360 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f2080009b80 tx=0x7f2080009190 comp rx=0 tx=0).stop 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 10 monclient: shutdown 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009eb10 msgr2=0x7f20900a3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 --2- 172.21.15.138:0/3668184414 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f209009eb10 0x7f20900a3c30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f20780ae040 tx=0x7f207801ab70 comp rx=0 tx=0).stop 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f2094a5c640 1 -- 172.21.15.138:0/3668184414 reap_dead start 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f2094a5c640 1 -- 172.21.15.138:0/3668184414 reap_dead start 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 shutdown_connections 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 >> 172.21.15.138:0/3668184414 conn(0x7f20900ab830 msgr2=0x7f20900a9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 shutdown_connections 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 -- 172.21.15.138:0/3668184414 wait complete. 2024-09-11T05:16:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.853+0000 7f20964bf640 1 librados: shutdown 2024-09-11T05:16:11.878 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:16:11.878 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:16:11.878 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:16:11.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 10 monclient: get_monmap_and_config 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 10 monclient: build_initial_monmap 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 10 monclient: monmap: 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:11.995994+0000 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:11.995994+0000 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:11.993 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:11.994 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:11.994 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:11.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 1 Processor -- start 2024-09-11T05:16:11.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 1 -- start start 2024-09-11T05:16:11.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 10 monclient: init 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f740a0a90 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.989+0000 7f7f7940b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f7409dc20 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f7940b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f7f7413abf0 0x7f7f7413cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f7940b640 10 monclient(hunting): picked mon.noname-a con 0x7f7f7413abf0 addr v2:172.21.15.138:3300/0 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f7940b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f7409efa0 con 0x7f7f7409dc20 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f7940b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f740a2d80 con 0x7f7f740a0a90 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f7940b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f7940b640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f7f740a37b0 con 0x7f7f7413abf0 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f7940b640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:11.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f7940b640 10 monclient(hunting): authenticate will time out at 1455.432062s 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f727fc640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f7f7409dc20 0x7f7f740a2670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:48900/0 (socket says 172.21.15.138:48900) 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f727fc640 1 -- 172.21.15.138:0/28931965 learned_addr learned my addr 172.21.15.138:0/28931965 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f7f7413abf0 0x7f7f7413cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient(hunting): get_auth_request con 0x7f7f7413abf0 auth_method 0 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient(hunting): handle_auth_done global_id 64434 payload 274 2024-09-11T05:16:11.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 1 -- 172.21.15.138:0/28931965 >> v1:172.21.15.190:6789/0 conn(0x7f7f740a0a90 legacy=0x7f7f740a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:11.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 1 -- 172.21.15.138:0/28931965 >> v1:172.21.15.138:6789/0 conn(0x7f7f7409dc20 legacy=0x7f7f740a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:11.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient: _finish_hunting 0 2024-09-11T05:16:11.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 1 monclient: found mon.noname-a 2024-09-11T05:16:11.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:11.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:11.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.993+0000 7f7f737fe640 1 -- 172.21.15.138:0/28931965 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7409ecc0 con 0x7f7f7413abf0 2024-09-11T05:16:12.002 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f737fe640 1 -- 172.21.15.138:0/28931965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f7413abf0 msgr2=0x7f7f7413cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f737fe640 1 --2- 172.21.15.138:0/28931965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f7413abf0 0x7f7f7413cfe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f640097c0 tx=0x7f7f6401c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7058757fc12618 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 1 -- 172.21.15.138:0/28931965 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f640b0070 con 0x7f7f7413abf0 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: dump: 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:12.003 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: _finish_auth 0 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 1 -- 172.21.15.138:0/28931965 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f640ab600 con 0x7f7f7413abf0 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 5 monclient: authenticate success, global_id 64434 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 1 -- 172.21.15.138:0/28931965 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f640ab9d0 con 0x7f7f7413abf0 2024-09-11T05:16:12.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 4 set_mon_vals no callback set 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f71ffb640 10 monclient: dump: 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:12.005 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient: shutdown 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- 172.21.15.138:0/28931965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f7413abf0 msgr2=0x7f7f7413cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 --2- 172.21.15.138:0/28931965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f7413abf0 0x7f7f7413cfe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f640097c0 tx=0x7f7f6401c180 comp rx=0 tx=0).stop 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- 172.21.15.138:0/28931965 shutdown_connections 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 --2- 172.21.15.138:0/28931965 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f7413abf0 0x7f7f7413cfe0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- 172.21.15.138:0/28931965 >> 172.21.15.138:0/28931965 conn(0x7f7f74063640 msgr2=0x7f7f74063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- 172.21.15.138:0/28931965 shutdown_connections 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- 172.21.15.138:0/28931965 wait complete. 2024-09-11T05:16:12.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient: build_initial_monmap 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient: monmap: 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 librados: starting msgr at 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 librados: starting objecter 2024-09-11T05:16:12.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 Processor -- start 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- start start 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 librados: setting wanted keys 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 librados: calling monclient init 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient: init 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f7409dc20 0x7f7f74120aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient(hunting): picked mon.noname-c con 0x7f7f7409dc20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f740a0a90 0x7f7f741ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient(hunting): picked mon.noname-a con 0x7f7f740a0a90 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f7413abf0 0x7f7f7411e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient(hunting): picked mon.noname-b con 0x7f7f7413abf0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7f741412e0 con 0x7f7f740a0a90 2024-09-11T05:16:12.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7f74141160 con 0x7f7f7413abf0 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7f74141460 con 0x7f7f7409dc20 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f7940b640 10 monclient(hunting): authenticate will time out at 1455.439085s 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f740a0a90 0x7f7f741ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f740a0a90 0x7f7f741ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:52012/0 (socket says 172.21.15.138:52012) 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f72ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f7409dc20 0x7f7f74120aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 -- 172.21.15.138:0/2484823080 learned_addr learned my addr 172.21.15.138:0/2484823080 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient(hunting): get_auth_request con 0x7f7f740a0a90 auth_method 0 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:12.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f72ffd640 10 monclient(hunting): get_auth_request con 0x7f7f7409dc20 auth_method 0 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:12.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient(hunting): handle_auth_done global_id 64437 payload 824 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f737fe640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f7413abf0 0x7f7f7411e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 -- 172.21.15.138:0/2484823080 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f7409dc20 msgr2=0x7f7f74120aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f7409dc20 0x7f7f74120aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 -- 172.21.15.138:0/2484823080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f7413abf0 msgr2=0x7f7f7411e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f7413abf0 0x7f7f7411e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient: _finish_hunting 0 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 monclient: found mon.noname-a 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f727fc640 1 -- 172.21.15.138:0/2484823080 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f74121ce0 con 0x7f7f740a0a90 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f737fe640 10 monclient: get_auth_request con 0x7f7f7413abf0 auth_method 0 2024-09-11T05:16:12.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:11.997+0000 7f7f737fe640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f7413abf0 0x7f7f7411e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:16:12.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f727fc640 1 -- 172.21.15.138:0/2484823080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f740a0a90 msgr2=0x7f7f741ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f727fc640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f740a0a90 0x7f7f741ceef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00cc10 tx=0x7f7f5c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 client.?.objecter ms_handle_connect 0x7f7f740a0a90 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 1 -- 172.21.15.138:0/2484823080 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c012400 con 0x7f7f740a0a90 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: dump: 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:12.014 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 1 -- 172.21.15.138:0/2484823080 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f5c0125e0 con 0x7f7f740a0a90 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 5 monclient: authenticate success, global_id 64437 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 10 monclient: _renew_subs 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f74121fc0 con 0x7f7f740a0a90 2024-09-11T05:16:12.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 1 -- 172.21.15.138:0/2484823080 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c0129d0 con 0x7f7f740a0a90 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 monclient: dump: 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:12.016 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f70ff9640 4 set_mon_vals no callback set 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 10 client.64437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 10 monclient: _renew_subs 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7409e110 con 0x7f7f740a0a90 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 1 librados: init done 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f740a2670 con 0x7f7f740a0a90 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 1 -- 172.21.15.138:0/2484823080 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f7f5c012bf0 con 0x7f7f740a0a90 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 1 --2- 172.21.15.138:0/2484823080 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7f5803cf20 0x7f7f5803f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 1 -- 172.21.15.138:0/2484823080 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (secure 0 0 0) 0x7f7f5c0500e0 con 0x7f7f740a0a90 2024-09-11T05:16:12.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 10 client.64437.objecter ms_dispatch 0x7f7f74000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 3 client.64437.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 3 client.64437.objecter handle_osd_map decoding full epoch 365 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f6b7fe640 20 client.64437.objecter dump_active .. 0 homeless 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f72ffd640 1 --2- 172.21.15.138:0/2484823080 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7f5803cf20 0x7f7f5803f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.001+0000 7f7f72ffd640 10 monclient: get_auth_request con 0x7f7f5803cf20 auth_method 0 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.005+0000 7f7f72ffd640 1 --2- 172.21.15.138:0/2484823080 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7f5803cf20 0x7f7f5803f3e0 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c007fd0 tx=0x7f7f6c007e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.005+0000 7f7f6b7fe640 10 client.64437.objecter ms_handle_connect 0x7f7f5803cf20 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.005+0000 7f7f6b7fe640 1 -- 172.21.15.138:0/2484823080 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f7f5c01bbb0 con 0x7f7f740a0a90 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.005+0000 7f7f6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:12.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.005+0000 7f7f6b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:12.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f7f7940b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:16:12.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f7f7940b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:16:12.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f7f7940b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:12.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7f7411ff40 con 0x7f7f740a0a90 2024-09-11T05:16:12.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f7f6b7fe640 1 -- 172.21.15.138:0/2484823080 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f7f5c01a360 con 0x7f7f740a0a90 2024-09-11T05:16:12.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f7f6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:16:12.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f7f6b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:12.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=3981332499,config=692619927,health=1995790572,kv=930862188,logm=3213995765}) 2024-09-11T05:16:12.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=745031329,config=692619927,health=1995790572,kv=1804961813,logm=495510221}) 2024-09-11T05:16:12.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.241+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=3981332499,config=692619927,health=1995790572,kv=930862188,logm=3213995765}) 2024-09-11T05:16:12.247 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=3981332499,config=2435484489,health=1995790572,kv=310889196,logm=523611020}) 2024-09-11T05:16:12.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 10 librados: watch_flush enter 2024-09-11T05:16:12.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 10 librados: watch_flush exit 2024-09-11T05:16:12.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 20 client.64437.objecter shutdown clearing up homeless session... 2024-09-11T05:16:12.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 10 client.64437.objecter successfully canceled tick 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7f5803cf20 msgr2=0x7f7f5803f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 --2- 172.21.15.138:0/2484823080 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7f5803cf20 0x7f7f5803f3e0 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c007fd0 tx=0x7f7f6c007e40 comp rx=0 tx=0).stop 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 10 monclient: shutdown 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f740a0a90 msgr2=0x7f7f741ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f740a0a90 0x7f7f741ceef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00cc10 tx=0x7f7f5c00a430 comp rx=0 tx=0).stop 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 shutdown_connections 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 --2- 172.21.15.138:0/2484823080 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7f5803cf20 0x7f7f5803f3e0 unknown :-1 s=CLOSED pgs=3422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f7413abf0 0x7f7f7411e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7f740a0a90 0x7f7f741ceef0 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00cc10 tx=0x7f7f5c00a430 comp rx=0 tx=0).stop 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 --2- 172.21.15.138:0/2484823080 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7f7409dc20 0x7f7f74120aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 >> 172.21.15.138:0/2484823080 conn(0x7f7f74063640 msgr2=0x7f7f740a6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:12.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 shutdown_connections 2024-09-11T05:16:12.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 -- 172.21.15.138:0/2484823080 wait complete. 2024-09-11T05:16:12.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f7f7940b640 1 librados: shutdown 2024-09-11T05:16:12.250 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.250 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908147982}) 2024-09-11T05:16:12.250 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=461986700}) 2024-09-11T05:16:12.251 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.251 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908147982}) 2024-09-11T05:16:12.251 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.245+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=553481681}) 2024-09-11T05:16:12.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.249+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.249+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258465878}) 2024-09-11T05:16:12.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.249+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=159375622}) 2024-09-11T05:16:12.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.249+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.249+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258465878}) 2024-09-11T05:16:12.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.249+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390133490}) 2024-09-11T05:16:12.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385313545}) 2024-09-11T05:16:12.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728176061}) 2024-09-11T05:16:12.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385313545}) 2024-09-11T05:16:12.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687823466}) 2024-09-11T05:16:12.256 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.256 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093806395}) 2024-09-11T05:16:12.256 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3225649190}) 2024-09-11T05:16:12.256 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.256 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093806395}) 2024-09-11T05:16:12.256 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424555495}) 2024-09-11T05:16:12.258 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.258 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.253+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340007162}) 2024-09-11T05:16:12.258 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1256320478}) 2024-09-11T05:16:12.258 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.258 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340007162}) 2024-09-11T05:16:12.258 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=471510591}) 2024-09-11T05:16:12.260 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.260 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479952689}) 2024-09-11T05:16:12.260 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093458924}) 2024-09-11T05:16:12.260 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.260 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479952689}) 2024-09-11T05:16:12.260 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.257+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4161522932}) 2024-09-11T05:16:12.262 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.262 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645655462}) 2024-09-11T05:16:12.262 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192753245}) 2024-09-11T05:16:12.262 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.262 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645655462}) 2024-09-11T05:16:12.262 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3030381807}) 2024-09-11T05:16:12.264 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.264 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039170502}) 2024-09-11T05:16:12.264 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4068065910}) 2024-09-11T05:16:12.264 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.264 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039170502}) 2024-09-11T05:16:12.264 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.261+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179923297}) 2024-09-11T05:16:12.266 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:16:12.267 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.267 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2549315924}) 2024-09-11T05:16:12.267 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792588000}) 2024-09-11T05:16:12.267 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.267 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2549315924}) 2024-09-11T05:16:12.267 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884583535}) 2024-09-11T05:16:12.268 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.268 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175687826}) 2024-09-11T05:16:12.268 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3507486069}) 2024-09-11T05:16:12.268 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.268 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175687826}) 2024-09-11T05:16:12.268 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.265+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657615439}) 2024-09-11T05:16:12.270 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.270 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785967063}) 2024-09-11T05:16:12.270 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532336159}) 2024-09-11T05:16:12.270 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.270 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785967063}) 2024-09-11T05:16:12.270 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758517864}) 2024-09-11T05:16:12.272 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.272 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735479437}) 2024-09-11T05:16:12.272 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3019932057}) 2024-09-11T05:16:12.273 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.273 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735479437}) 2024-09-11T05:16:12.273 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.269+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=151946634}) 2024-09-11T05:16:12.274 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.274 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522657033}) 2024-09-11T05:16:12.274 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148555691}) 2024-09-11T05:16:12.275 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.275 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522657033}) 2024-09-11T05:16:12.275 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=131352707}) 2024-09-11T05:16:12.276 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.277 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064937888}) 2024-09-11T05:16:12.277 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=890003550}) 2024-09-11T05:16:12.277 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.277 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064937888}) 2024-09-11T05:16:12.277 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.273+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207221524}) 2024-09-11T05:16:12.278 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.278 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193612137}) 2024-09-11T05:16:12.278 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430989323}) 2024-09-11T05:16:12.278 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.278 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193612137}) 2024-09-11T05:16:12.279 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=368357966}) 2024-09-11T05:16:12.280 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.280 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158571208}) 2024-09-11T05:16:12.281 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=917734259}) 2024-09-11T05:16:12.281 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.281 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158571208}) 2024-09-11T05:16:12.281 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.277+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317273283}) 2024-09-11T05:16:12.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365633031}) 2024-09-11T05:16:12.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1019065823}) 2024-09-11T05:16:12.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365633031}) 2024-09-11T05:16:12.843 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.841+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697787129}) 2024-09-11T05:16:12.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.901+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.901+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967468728}) 2024-09-11T05:16:12.902 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.901+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076342498}) 2024-09-11T05:16:12.902 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.901+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.902 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.901+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967468728}) 2024-09-11T05:16:12.902 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.901+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367910343}) 2024-09-11T05:16:12.909 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.905+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.909 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.905+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839217400}) 2024-09-11T05:16:12.909 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.905+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755881304}) 2024-09-11T05:16:12.909 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.905+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.909 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.905+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839217400}) 2024-09-11T05:16:12.909 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.905+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269472795}) 2024-09-11T05:16:12.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.921+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.921+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590769098}) 2024-09-11T05:16:12.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.921+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2029036340}) 2024-09-11T05:16:12.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.921+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:12.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.921+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590769098}) 2024-09-11T05:16:12.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:12.921+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=859724643}) 2024-09-11T05:16:13.141 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.141 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543385460}) 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1224980644}) 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543385460}) 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815394235}) 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3856838245}) 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1703674779}) 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3856838245}) 2024-09-11T05:16:13.142 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.137+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273684429}) 2024-09-11T05:16:13.144 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.141+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.144 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.141+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226393771}) 2024-09-11T05:16:13.144 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.141+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=706837734}) 2024-09-11T05:16:13.144 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.141+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.144 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.141+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226393771}) 2024-09-11T05:16:13.144 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.141+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=32609372}) 2024-09-11T05:16:13.146 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.146 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796755665}) 2024-09-11T05:16:13.146 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=815944687}) 2024-09-11T05:16:13.147 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.147 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796755665}) 2024-09-11T05:16:13.147 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792756543}) 2024-09-11T05:16:13.148 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.149 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3984392285}) 2024-09-11T05:16:13.149 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517036848}) 2024-09-11T05:16:13.149 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.149 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3984392285}) 2024-09-11T05:16:13.149 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.145+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2205289783}) 2024-09-11T05:16:13.151 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.151 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3518570170}) 2024-09-11T05:16:13.151 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=31351653}) 2024-09-11T05:16:13.151 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.151 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3518570170}) 2024-09-11T05:16:13.151 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3979309466}) 2024-09-11T05:16:13.154 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.154 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722274891}) 2024-09-11T05:16:13.154 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480716955}) 2024-09-11T05:16:13.154 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.154 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722274891}) 2024-09-11T05:16:13.154 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.149+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1453380308}) 2024-09-11T05:16:13.156 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.153+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.156 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.153+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854174835}) 2024-09-11T05:16:13.156 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.153+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1104990155}) 2024-09-11T05:16:13.156 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.153+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.156 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.153+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854174835}) 2024-09-11T05:16:13.156 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.153+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910883372}) 2024-09-11T05:16:13.158 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.158 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456432494}) 2024-09-11T05:16:13.158 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893367168}) 2024-09-11T05:16:13.158 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.158 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456432494}) 2024-09-11T05:16:13.159 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3060228323}) 2024-09-11T05:16:13.161 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.161 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258653537}) 2024-09-11T05:16:13.161 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914379377}) 2024-09-11T05:16:13.161 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.161 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258653537}) 2024-09-11T05:16:13.161 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.157+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3071407375}) 2024-09-11T05:16:13.164 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.161+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.164 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.161+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619948543}) 2024-09-11T05:16:13.164 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.161+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769918426}) 2024-09-11T05:16:13.164 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.161+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.164 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.161+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619948543}) 2024-09-11T05:16:13.164 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.161+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061852332}) 2024-09-11T05:16:13.166 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.161+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.166 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.161+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983131893}) 2024-09-11T05:16:13.166 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1789798433}) 2024-09-11T05:16:13.166 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.166 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983131893}) 2024-09-11T05:16:13.166 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=212103505}) 2024-09-11T05:16:13.168 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.168 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440121039}) 2024-09-11T05:16:13.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3137656342}) 2024-09-11T05:16:13.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440121039}) 2024-09-11T05:16:13.169 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.165+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450253247}) 2024-09-11T05:16:13.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774798805}) 2024-09-11T05:16:13.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851986520}) 2024-09-11T05:16:13.171 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.172 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774798805}) 2024-09-11T05:16:13.172 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1361967489}) 2024-09-11T05:16:13.173 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847071765}) 2024-09-11T05:16:13.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=363064421}) 2024-09-11T05:16:13.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847071765}) 2024-09-11T05:16:13.174 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.169+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2209202817}) 2024-09-11T05:16:13.175 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.173+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.175 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.173+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629349011}) 2024-09-11T05:16:13.175 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.173+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957477761}) 2024-09-11T05:16:13.175 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.173+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.176 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.173+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629349011}) 2024-09-11T05:16:13.176 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.173+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=650566444}) 2024-09-11T05:16:13.178 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=16} crc {logm=4031496379,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=567410791,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=1287100665,osd_snap=2344384637,osdmap=943326310}) 2024-09-11T05:16:13.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=18} crc {logm=1304427165,mdsmap=1754867906,mgr=2081689912,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=498402237,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=1287100665,osd_snap=3062036323,osdmap=3106765596}) 2024-09-11T05:16:13.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=16} crc {logm=4031496379,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=567410791,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=1287100665,osd_snap=2344384637,osdmap=943326310}) 2024-09-11T05:16:13.179 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=5,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=20} crc {logm=3703596032,mdsmap=78050798,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=567410791,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=1287100665,osd_snap=1399733834,osdmap=3150976563}) 2024-09-11T05:16:13.181 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=439574416}) 2024-09-11T05:16:13.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=724903270}) 2024-09-11T05:16:13.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=439574416}) 2024-09-11T05:16:13.182 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.177+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3595795696}) 2024-09-11T05:16:13.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.181+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.181+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1689300291}) 2024-09-11T05:16:13.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.181+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2148798188}) 2024-09-11T05:16:13.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.181+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.181+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1689300291}) 2024-09-11T05:16:13.185 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.181+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2686467132}) 2024-09-11T05:16:13.188 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.185+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.188 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.185+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2251720999}) 2024-09-11T05:16:13.188 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.185+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1201118991}) 2024-09-11T05:16:13.188 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.185+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.188 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.185+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2251720999}) 2024-09-11T05:16:13.188 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.185+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3883702862}) 2024-09-11T05:16:13.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.189+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.189+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=585765423}) 2024-09-11T05:16:13.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.189+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2172120968}) 2024-09-11T05:16:13.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.189+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.189+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=585765423}) 2024-09-11T05:16:13.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.189+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3154684212}) 2024-09-11T05:16:13.197 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.193+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.197 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.193+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3442955956}) 2024-09-11T05:16:13.197 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.193+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2898449858}) 2024-09-11T05:16:13.197 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.193+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.197 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.193+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3442955956}) 2024-09-11T05:16:13.197 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.193+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3632271098}) 2024-09-11T05:16:13.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.209+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.209+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=25} crc {osdmap=1543876447}) 2024-09-11T05:16:13.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.209+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=25} crc {osdmap=3552833388}) 2024-09-11T05:16:13.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.209+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:16:13.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.209+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=25} crc {osdmap=1543876447}) 2024-09-11T05:16:13.213 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:16:13.209+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=25} crc {osdmap=876297230}) 2024-09-11T05:16:13.267 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:16:13.267 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:16:13.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient: get_monmap_and_config 2024-09-11T05:16:13.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient: build_initial_monmap 2024-09-11T05:16:13.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient: monmap: 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:13.402698+0000 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:13.402698+0000 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:13.400 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 1 Processor -- start 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 1 -- start start 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient: init 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb05006e0e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009eb30 0x7fb05013a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient(hunting): picked mon.noname-b con 0x7fb05009eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb05009ef10 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:13.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb05006e4c0 con 0x7fb05006e0e0 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb05009f540 con 0x7fb05009ef10 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb05009f820 con 0x7fb05009eb30 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb057b6c640 10 monclient(hunting): authenticate will time out at 1456.838886s 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009eb30 0x7fb05013a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009eb30 0x7fb05013a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:56910/0 (socket says 172.21.15.138:56910) 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 1 -- 172.21.15.138:0/3855731969 learned_addr learned my addr 172.21.15.138:0/3855731969 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient(hunting): get_auth_request con 0x7fb05009eb30 auth_method 0 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient(hunting): handle_auth_done global_id 84116 payload 274 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 1 -- 172.21.15.138:0/3855731969 >> v1:172.21.15.190:6789/0 conn(0x7fb05009ef10 legacy=0x7fb05013ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 1 -- 172.21.15.138:0/3855731969 >> v1:172.21.15.138:6789/0 conn(0x7fb05006e0e0 legacy=0x7fb05009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient: _finish_hunting 0 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 1 monclient: found mon.noname-b 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:16:13.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.397+0000 7fb055107640 1 -- 172.21.15.138:0/3855731969 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05009fb00 con 0x7fb05009eb30 2024-09-11T05:16:13.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb055107640 1 -- 172.21.15.138:0/3855731969 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009eb30 msgr2=0x7fb05013a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb055107640 1 --2- 172.21.15.138:0/3855731969 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009eb30 0x7fb05013a930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb04400d180 tx=0x7fb04401eab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf6cce5d03105113 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 1 -- 172.21.15.138:0/3855731969 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04400b3b0 con 0x7fb05009eb30 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: dump: 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.407 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: _finish_auth 0 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: _check_auth_tickets 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 1 -- 172.21.15.138:0/3855731969 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0440a7740 con 0x7fb05009eb30 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb057b6c640 5 monclient: authenticate success, global_id 84116 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb057b6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:13.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb057b6c640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 1 -- 172.21.15.138:0/3855731969 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0440a7b10 con 0x7fb05009eb30 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb057b6c640 4 set_mon_vals no callback set 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.401+0000 7fb054906640 10 monclient: dump: 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.409 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient: shutdown 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- 172.21.15.138:0/3855731969 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009eb30 msgr2=0x7fb05013a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 --2- 172.21.15.138:0/3855731969 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009eb30 0x7fb05013a930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb04400d180 tx=0x7fb04401eab0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- 172.21.15.138:0/3855731969 shutdown_connections 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 --2- 172.21.15.138:0/3855731969 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009eb30 0x7fb05013a930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- 172.21.15.138:0/3855731969 >> 172.21.15.138:0/3855731969 conn(0x7fb050063640 msgr2=0x7fb050063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- 172.21.15.138:0/3855731969 shutdown_connections 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- 172.21.15.138:0/3855731969 wait complete. 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient: build_initial_monmap 2024-09-11T05:16:13.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient: monmap: 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:13.411 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 librados: starting msgr at 2024-09-11T05:16:13.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 librados: starting objecter 2024-09-11T05:16:13.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 Processor -- start 2024-09-11T05:16:13.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- start start 2024-09-11T05:16:13.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 librados: setting wanted keys 2024-09-11T05:16:13.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 librados: calling monclient init 2024-09-11T05:16:13.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient: init 2024-09-11T05:16:13.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb05006e0e0 0x7fb050135160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient(hunting): picked mon.noname-b con 0x7fb05006e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb05009eb30 0x7fb0501356a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient(hunting): picked mon.noname-a con 0x7fb05009eb30 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009ef10 0x7fb05013b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient(hunting): picked mon.noname-c con 0x7fb05009ef10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0500a2d50 con 0x7fb05009eb30 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb0500a2bd0 con 0x7fb05006e0e0 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb0500a2ed0 con 0x7fb05009ef10 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb057b6c640 10 monclient(hunting): authenticate will time out at 1456.846053s 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009ef10 0x7fb05013b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb05009eb30 0x7fb0501356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb05009eb30 0x7fb0501356a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:52018/0 (socket says 172.21.15.138:52018) 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 -- 172.21.15.138:0/1431665087 learned_addr learned my addr 172.21.15.138:0/1431665087 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:13.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient(hunting): get_auth_request con 0x7fb05009eb30 auth_method 0 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009ef10 0x7fb05013b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:56916/0 (socket says 172.21.15.138:56916) 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 10 monclient(hunting): get_auth_request con 0x7fb05009ef10 auth_method 0 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055908640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb05006e0e0 0x7fb050135160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055908640 10 monclient(hunting): get_auth_request con 0x7fb05006e0e0 auth_method 0 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055908640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055908640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient(hunting): handle_auth_done global_id 64440 payload 824 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 -- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009ef10 msgr2=0x7fb05013b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009ef10 0x7fb05013b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 -- 172.21.15.138:0/1431665087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb05006e0e0 msgr2=0x7fb050135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb05006e0e0 0x7fb050135160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient: _finish_hunting 0 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 monclient: found mon.noname-a 2024-09-11T05:16:13.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb055107640 1 -- 172.21.15.138:0/1431665087 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb050070370 con 0x7fb05009eb30 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.405+0000 7fb056109640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009ef10 0x7fb05013b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb055107640 1 -- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb05009eb30 msgr2=0x7fb0501356a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb055107640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb05009eb30 0x7fb0501356a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb0440b1040 tx=0x7fb04401b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 client.?.objecter ms_handle_connect 0x7fb05009eb30 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 1 -- 172.21.15.138:0/1431665087 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0440be020 con 0x7fb05009eb30 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: dump: 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.421 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: _finish_auth 0 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: _check_auth_tickets 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 1 -- 172.21.15.138:0/1431665087 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb044009c60 con 0x7fb05009eb30 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 5 monclient: authenticate success, global_id 64440 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 10 monclient: _renew_subs 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05011ed10 con 0x7fb05009eb30 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 1 -- 172.21.15.138:0/1431665087 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0440ba6b0 con 0x7fb05009eb30 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 monclient: dump: 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.423 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0437fe640 4 set_mon_vals no callback set 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 10 client.64440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 10 monclient: _renew_subs 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05009e110 con 0x7fb05009eb30 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 1 librados: init done 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05006f4b0 con 0x7fb05009eb30 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 1 -- 172.21.15.138:0/1431665087 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fb0440ba850 con 0x7fb05009eb30 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 1 --2- 172.21.15.138:0/1431665087 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb03403ced0 0x7fb03403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 1 -- 172.21.15.138:0/1431665087 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (secure 0 0 0) 0x7fb0440ed0b0 con 0x7fb05009eb30 2024-09-11T05:16:13.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 10 client.64440.objecter ms_dispatch 0x7fb050000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 3 client.64440.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 3 client.64440.objecter handle_osd_map decoding full epoch 365 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb0427fc640 20 client.64440.objecter dump_active .. 0 homeless 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb055908640 1 --2- 172.21.15.138:0/1431665087 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb03403ced0 0x7fb03403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.409+0000 7fb055908640 10 monclient: get_auth_request con 0x7fb03403ced0 auth_method 0 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.413+0000 7fb055908640 1 --2- 172.21.15.138:0/1431665087 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb03403ced0 0x7fb03403f390 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00b810 tx=0x7fb04c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.413+0000 7fb0427fc640 10 client.64440.objecter ms_handle_connect 0x7fb03403ced0 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.413+0000 7fb0427fc640 1 -- 172.21.15.138:0/1431665087 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fb0440c8030 con 0x7fb05009eb30 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.413+0000 7fb0427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:13.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.413+0000 7fb0427fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:13.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.645+0000 7fb057b6c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:16:13.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.645+0000 7fb057b6c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:16:13.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.645+0000 7fb057b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.645+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb0500a8010 con 0x7fb05009eb30 2024-09-11T05:16:13.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb0427fc640 1 -- 172.21.15.138:0/1431665087 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fb044018820 con 0x7fb05009eb30 2024-09-11T05:16:13.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb0427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:16:13.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb0427fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:13.650 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:16:13.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 10 librados: watch_flush enter 2024-09-11T05:16:13.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 10 librados: watch_flush exit 2024-09-11T05:16:13.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 20 client.64440.objecter shutdown clearing up homeless session... 2024-09-11T05:16:13.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 10 client.64440.objecter successfully canceled tick 2024-09-11T05:16:13.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb03403ced0 msgr2=0x7fb03403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 --2- 172.21.15.138:0/1431665087 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb03403ced0 0x7fb03403f390 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00b810 tx=0x7fb04c008040 comp rx=0 tx=0).stop 2024-09-11T05:16:13.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 10 monclient: shutdown 2024-09-11T05:16:13.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb05009eb30 msgr2=0x7fb0501356a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb05009eb30 0x7fb0501356a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb0440b1040 tx=0x7fb04401b8b0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 shutdown_connections 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 --2- 172.21.15.138:0/1431665087 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb03403ced0 0x7fb03403f390 unknown :-1 s=CLOSED pgs=3423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb05009ef10 0x7fb05013b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb05009eb30 0x7fb0501356a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 --2- 172.21.15.138:0/1431665087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb05006e0e0 0x7fb050135160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 >> 172.21.15.138:0/1431665087 conn(0x7fb050063640 msgr2=0x7fb050072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 shutdown_connections 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 -- 172.21.15.138:0/1431665087 wait complete. 2024-09-11T05:16:13.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.649+0000 7fb057b6c640 1 librados: shutdown 2024-09-11T05:16:13.668 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:16:13.668 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:16:13.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:16:13.668 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.781+0000 7f0bda1f5640 10 monclient: get_monmap_and_config 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.781+0000 7f0bda1f5640 10 monclient: build_initial_monmap 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.781+0000 7f0bda1f5640 10 monclient: monmap: 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:13.789200+0000 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:13.789200+0000 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:13.786 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:13.787 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:13.787 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:13.787 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:13.787 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 1 Processor -- start 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 1 -- start start 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient: init 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0bd40a32c0 0x7f0bd409e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd40a32c0 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd409e960 0x7f0bd409ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient(hunting): picked mon.noname-b con 0x7f0bd409e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bd41356f0 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bd40a1b50 con 0x7f0bd41356f0 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0bd409fe70 con 0x7f0bd40a32c0 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0bd40a1920 con 0x7f0bd409e960 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bda1f5640 10 monclient(hunting): authenticate will time out at 1457.225248s 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd3fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f0bd41356f0 0x7f0bd4135b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:48922/0 (socket says 172.21.15.138:48922) 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd409e960 0x7f0bd409ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd3fff640 1 -- 172.21.15.138:0/3058427645 learned_addr learned my addr 172.21.15.138:0/3058427645 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f0bd409e960 auth_method 0 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd37fe640 1 --2- 172.21.15.138:0/3058427645 >> v2:172.21.15.190:3300/0 conn(0x7f0bd40a32c0 0x7f0bd409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient(hunting): handle_auth_done global_id 84122 payload 274 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 1 -- 172.21.15.138:0/3058427645 >> v2:172.21.15.190:3300/0 conn(0x7f0bd40a32c0 msgr2=0x7f0bd409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 1 --2- 172.21.15.138:0/3058427645 >> v2:172.21.15.190:3300/0 conn(0x7f0bd40a32c0 0x7f0bd409e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 1 -- 172.21.15.138:0/3058427645 >> v1:172.21.15.138:6789/0 conn(0x7f0bd41356f0 legacy=0x7f0bd4135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 1 monclient: found mon.noname-b 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd2ffd640 1 -- 172.21.15.138:0/3058427645 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd409f560 con 0x7f0bd409e960 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd37fe640 10 monclient: get_auth_request con 0x7f0bd40a32c0 auth_method 0 2024-09-11T05:16:13.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.785+0000 7f0bd37fe640 1 --2- 172.21.15.138:0/3058427645 >> v2:172.21.15.190:3300/0 conn(0x7f0bd40a32c0 0x7f0bd409e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:16:13.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd2ffd640 1 -- 172.21.15.138:0/3058427645 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd409e960 msgr2=0x7f0bd409ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd2ffd640 1 --2- 172.21.15.138:0/3058427645 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd409e960 0x7f0bd409ed80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80097c0 tx=0x7f0bc801c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cd33d6f0d80405d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 1 -- 172.21.15.138:0/3058427645 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc80b0070 con 0x7f0bd409e960 2024-09-11T05:16:13.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: dump: 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: _finish_auth 0 2024-09-11T05:16:13.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: _check_auth_tickets 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 1 -- 172.21.15.138:0/3058427645 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc80ab600 con 0x7f0bd409e960 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 1 -- 172.21.15.138:0/3058427645 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc80ab9d0 con 0x7f0bd409e960 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 5 monclient: authenticate success, global_id 84122 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 4 set_mon_vals no callback set 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bd27fc640 10 monclient: dump: 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.797 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 10 monclient: shutdown 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 -- 172.21.15.138:0/3058427645 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd409e960 msgr2=0x7f0bd409ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 --2- 172.21.15.138:0/3058427645 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd409e960 0x7f0bd409ed80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80097c0 tx=0x7f0bc801c390 comp rx=0 tx=0).stop 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 -- 172.21.15.138:0/3058427645 shutdown_connections 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 --2- 172.21.15.138:0/3058427645 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd409e960 0x7f0bd409ed80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 --2- 172.21.15.138:0/3058427645 >> v2:172.21.15.190:3300/0 conn(0x7f0bd40a32c0 0x7f0bd409e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 -- 172.21.15.138:0/3058427645 >> 172.21.15.138:0/3058427645 conn(0x7f0bd4063640 msgr2=0x7f0bd4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:13.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 -- 172.21.15.138:0/3058427645 shutdown_connections 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 -- 172.21.15.138:0/3058427645 wait complete. 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 10 monclient: build_initial_monmap 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 10 monclient: monmap: 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.799 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 librados: starting msgr at 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 librados: starting objecter 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 Processor -- start 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 -- start start 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 librados: setting wanted keys 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 1 librados: calling monclient init 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.789+0000 7f0bda1f5640 10 monclient: init 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:16:13.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bd409e960 0x7f0bd4072ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): picked mon.noname-b con 0x7f0bd409e960 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd40a32c0 0x7f0bd413c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): picked mon.noname-a con 0x7f0bd40a32c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd41356f0 0x7f0bd413c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd41356f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd40a12d0 con 0x7f0bd40a32c0 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd40a21b0 con 0x7f0bd409e960 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0bd4073ef0 con 0x7f0bd41356f0 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): authenticate will time out at 1457.232397s 2024-09-11T05:16:13.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd40a32c0 0x7f0bd413c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd41356f0 0x7f0bd413c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd41356f0 0x7f0bd413c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:56944/0 (socket says 172.21.15.138:56944) 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd40a32c0 0x7f0bd413c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:52032/0 (socket says 172.21.15.138:52032) 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 -- 172.21.15.138:0/2192015057 learned_addr learned my addr 172.21.15.138:0/2192015057 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient(hunting): get_auth_request con 0x7f0bd41356f0 auth_method 0 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f0bd40a32c0 auth_method 0 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd37fe640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bd409e960 0x7f0bd4072ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient(hunting): handle_auth_done global_id 84125 payload 824 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bd409e960 msgr2=0x7f0bd4072ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bd409e960 0x7f0bd4072ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd40a32c0 msgr2=0x7f0bd413c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd40a32c0 0x7f0bd413c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient: _finish_hunting 0 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 monclient: found mon.noname-c 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:16:13.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 -- 172.21.15.138:0/2192015057 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd4139b40 con 0x7f0bd41356f0 2024-09-11T05:16:13.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd37fe640 10 monclient: get_auth_request con 0x7f0bd409e960 auth_method 0 2024-09-11T05:16:13.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd37fe640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bd409e960 0x7f0bd4072ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:16:13.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd2ffd640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd40a32c0 0x7f0bd413c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd41356f0 msgr2=0x7f0bd413c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd3fff640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd41356f0 0x7f0bd413c5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00caf0 tx=0x7f0bcc00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 client.?.objecter ms_handle_connect 0x7f0bd41356f0 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc00ae50 con 0x7f0bd41356f0 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: dump: 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.808 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: _finish_auth 0 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: _check_auth_tickets 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bcc012400 con 0x7f0bd41356f0 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 5 monclient: authenticate success, global_id 84125 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient: _renew_subs 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd4137a40 con 0x7f0bd41356f0 2024-09-11T05:16:13.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0127d0 con 0x7f0bd41356f0 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 client.84125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bbbfff640 10 monclient: dump: 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.810 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient: _renew_subs 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd4136910 con 0x7f0bd41356f0 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 librados: init done 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bd1ffb640 4 set_mon_vals no callback set 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd4136ba0 0x7f0bd4136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.793+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd41394e0 con 0x7f0bd4136ba0 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd4136ba0 0x7f0bd4136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 10 monclient: get_auth_request con 0x7f0bd4136ba0 auth_method 0 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f0bcc012a30 con 0x7f0bd41356f0 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 10 monclient(hunting): handle_auth_done global_id 64446 payload 824 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 1 -- 172.21.15.138:0/2192015057 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bd409e110 con 0x7f0bd4136ba0 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 --2- 172.21.15.138:0/2192015057 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0bc003d070 0x7f0bc003f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (secure 0 0 0) 0x7f0bcc050060 con 0x7f0bd41356f0 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 client.84125.objecter ms_dispatch 0x7f0bd4000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 3 client.84125.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 3 client.84125.objecter handle_osd_map decoding full epoch 365 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 20 client.84125.objecter dump_active .. 0 homeless 2024-09-11T05:16:13.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd2ffd640 1 --2- 172.21.15.138:0/2192015057 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0bc003d070 0x7f0bc003f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd2ffd640 10 monclient: get_auth_request con 0x7f0bc003d070 auth_method 0 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 client.84125.objecter ms_handle_connect 0x7f0bd4136ba0 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd4136ba0 msgr2=0x7f0bd4136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 client.84125.objecter resend_mon_ops 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd37fe640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd4136ba0 0x7f0bd4136f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d790 tx=0x7f0bc400dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4016070 con 0x7f0bd4136ba0 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 monclient: dump: 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.813 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0bc4008b20 con 0x7f0bd4136ba0 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd4136ba0 msgr2=0x7f0bd4136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd4136ba0 0x7f0bd4136f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d790 tx=0x7f0bc400dc50 comp rx=0 tx=0).stop 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd4136ba0 msgr2=0x7f0bd4136f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd4136ba0 0x7f0bd4136f80 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d790 tx=0x7f0bc400dc50 comp rx=0 tx=0).stop 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bbbfff640 10 client.84125.objecter ms_handle_connect 0x7f0bc003d070 2024-09-11T05:16:13.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.797+0000 7f0bd2ffd640 1 --2- 172.21.15.138:0/2192015057 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0bc003d070 0x7f0bc003f530 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7f0bd406cfe0 tx=0x7f0bc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bda1f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bda1f5640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd41374c0 0x7f0bd406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bda1f5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd40a1450 con 0x7f0bd41374c0 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd41374c0 0x7f0bd406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 10 monclient: get_auth_request con 0x7f0bd41374c0 auth_method 0 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 10 monclient(hunting): handle_auth_done global_id 64449 payload 824 2024-09-11T05:16:13.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 1 -- 172.21.15.138:0/2192015057 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0bd40a2b60 con 0x7f0bd41374c0 2024-09-11T05:16:13.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd41374c0 msgr2=0x7f0bd406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 10 client.84125.objecter ms_handle_connect 0x7f0bd41374c0 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 10 client.84125.objecter resend_mon_ops 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bd37fe640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd41374c0 0x7f0bd406f2e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d790 tx=0x7f0bc4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc401b860 con 0x7f0bd41374c0 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 10 monclient: dump: 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.817 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f0bc40113c0 con 0x7f0bd41374c0 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd41374c0 msgr2=0x7f0bd406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd41374c0 0x7f0bd406f2e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d790 tx=0x7f0bc4004740 comp rx=0 tx=0).stop 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd41374c0 msgr2=0x7f0bd406f2e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bbbfff640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0bd41374c0 0x7f0bd406f2e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.818 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 32, 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 277435, 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 7, 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:16:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:16:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:16:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:16:13.830 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:16:13.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bda1f5640 10 librados: watch_flush enter 2024-09-11T05:16:13.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.805+0000 7f0bda1f5640 10 librados: watch_flush exit 2024-09-11T05:16:13.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 20 client.84125.objecter shutdown clearing up homeless session... 2024-09-11T05:16:13.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 10 client.84125.objecter successfully canceled tick 2024-09-11T05:16:13.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0bc003d070 msgr2=0x7f0bc003f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 --2- 172.21.15.138:0/2192015057 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0bc003d070 0x7f0bc003f530 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7f0bd406cfe0 tx=0x7f0bc8008040 comp rx=0 tx=0).stop 2024-09-11T05:16:13.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 10 monclient: shutdown 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd41356f0 msgr2=0x7f0bd413c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd41356f0 0x7f0bd413c5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00caf0 tx=0x7f0bcc00a060 comp rx=0 tx=0).stop 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bd3fff640 1 -- 172.21.15.138:0/2192015057 reap_dead start 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 shutdown_connections 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 --2- 172.21.15.138:0/2192015057 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0bd41356f0 0x7f0bd413c5c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 >> 172.21.15.138:0/2192015057 conn(0x7f0bd4063640 msgr2=0x7f0bd40a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 shutdown_connections 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 -- 172.21.15.138:0/2192015057 wait complete. 2024-09-11T05:16:13.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.809+0000 7f0bda1f5640 1 librados: shutdown 2024-09-11T05:16:13.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:16:13.834 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:16:13.834 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:16:13.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.937+0000 7f752e84b640 10 monclient: get_monmap_and_config 2024-09-11T05:16:13.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.937+0000 7f752e84b640 10 monclient: build_initial_monmap 2024-09-11T05:16:13.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.937+0000 7f752e84b640 10 monclient: monmap: 2024-09-11T05:16:13.941 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:13.941 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.941 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:13.944847+0000 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:13.944847+0000 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:13.942 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 1 Processor -- start 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 1 -- start start 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient: init 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7528070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f752809e6e0 0x7f752813d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient(hunting): picked mon.noname-a con 0x7f752809e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f752809eb10 0x7f752813fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient(hunting): picked mon.noname-c con 0x7f752809eb10 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7528135b60 con 0x7f7528070010 2024-09-11T05:16:13.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f75280703f0 con 0x7f752809e6e0 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f75280a8990 con 0x7f752809eb10 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752e84b640 10 monclient(hunting): authenticate will time out at 1457.380689s 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f752809e6e0 0x7f752813d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f752809e6e0 0x7f752813d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:52054/0 (socket says 172.21.15.138:52054) 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 1 -- 172.21.15.138:0/806076449 learned_addr learned my addr 172.21.15.138:0/806076449 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient(hunting): get_auth_request con 0x7f752809e6e0 auth_method 0 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752cde8640 1 --2- 172.21.15.138:0/806076449 >> v2:172.21.15.190:3300/0 conn(0x7f752809eb10 0x7f752813fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient(hunting): handle_auth_done global_id 64452 payload 274 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 1 -- 172.21.15.138:0/806076449 >> v2:172.21.15.190:3300/0 conn(0x7f752809eb10 msgr2=0x7f752813fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 1 --2- 172.21.15.138:0/806076449 >> v2:172.21.15.190:3300/0 conn(0x7f752809eb10 0x7f752813fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 1 -- 172.21.15.138:0/806076449 >> v1:172.21.15.190:6789/0 conn(0x7f7528070010 legacy=0x7f7528135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient: _finish_hunting 0 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 1 monclient: found mon.noname-a 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f75277fe640 1 -- 172.21.15.138:0/806076449 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752809cbf0 con 0x7f752809e6e0 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752cde8640 10 monclient: get_auth_request con 0x7f752809eb10 auth_method 0 2024-09-11T05:16:13.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.941+0000 7f752cde8640 1 --2- 172.21.15.138:0/806076449 >> v2:172.21.15.190:3300/0 conn(0x7f752809eb10 0x7f752813fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:16:13.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f75277fe640 1 -- 172.21.15.138:0/806076449 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f752809e6e0 msgr2=0x7f752813d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f75277fe640 1 --2- 172.21.15.138:0/806076449 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f752809e6e0 0x7f752813d640 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7528135200 tx=0x7f751c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7bd990b0248506 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 1 -- 172.21.15.138:0/806076449 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0b0070 con 0x7f752809e6e0 2024-09-11T05:16:13.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:13.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: dump: 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: _finish_auth 0 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:13.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 1 -- 172.21.15.138:0/806076449 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751c0ab600 con 0x7f752809e6e0 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 5 monclient: authenticate success, global_id 64452 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 1 -- 172.21.15.138:0/806076449 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0ab9d0 con 0x7f752809e6e0 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f7526ffd640 10 monclient: dump: 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.953 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 4 set_mon_vals no callback set 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 10 monclient: shutdown 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 -- 172.21.15.138:0/806076449 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f752809e6e0 msgr2=0x7f752813d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 --2- 172.21.15.138:0/806076449 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f752809e6e0 0x7f752813d640 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7528135200 tx=0x7f751c01c3b0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 -- 172.21.15.138:0/806076449 shutdown_connections 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 --2- 172.21.15.138:0/806076449 >> v2:172.21.15.190:3300/0 conn(0x7f752809eb10 0x7f752813fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 --2- 172.21.15.138:0/806076449 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f752809e6e0 0x7f752813d640 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 -- 172.21.15.138:0/806076449 >> 172.21.15.138:0/806076449 conn(0x7f75280ab830 msgr2=0x7f75280a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 -- 172.21.15.138:0/806076449 shutdown_connections 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 -- 172.21.15.138:0/806076449 wait complete. 2024-09-11T05:16:13.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 10 monclient: build_initial_monmap 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 10 monclient: monmap: 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 librados: starting msgr at 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 librados: starting objecter 2024-09-11T05:16:13.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 Processor -- start 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 -- start start 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 librados: setting wanted keys 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 1 librados: calling monclient init 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.945+0000 7f752e84b640 10 monclient: init 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7528070010 0x7f75281d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): picked mon.noname-a con 0x7f7528070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f752809e6e0 0x7f752813f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): picked mon.noname-b con 0x7f752809e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752809eb10 0x7f75280a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): picked mon.noname-c con 0x7f752809eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7528142c80 con 0x7f7528070010 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7528142b00 con 0x7f752809e6e0 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7528142e00 con 0x7f752809eb10 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): authenticate will time out at 1457.388179s 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7528070010 0x7f75281d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752cde8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752809eb10 0x7f75280a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7528070010 0x7f75281d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:52066/0 (socket says 172.21.15.138:52066) 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 -- 172.21.15.138:0/1630668638 learned_addr learned my addr 172.21.15.138:0/1630668638 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient(hunting): get_auth_request con 0x7f7528070010 auth_method 0 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752cde8640 10 monclient(hunting): get_auth_request con 0x7f752809eb10 auth_method 0 2024-09-11T05:16:13.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752cde8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752cde8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752cde8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752cde8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752cde8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752cde8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient(hunting): handle_auth_done global_id 64455 payload 824 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752809eb10 msgr2=0x7f75280a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752809eb10 0x7f75280a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f752809e6e0 msgr2=0x7f752813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f752809e6e0 0x7f752813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient: _finish_hunting 0 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 monclient: found mon.noname-a 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:13.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 -- 172.21.15.138:0/1630668638 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75281336c0 con 0x7f7528070010 2024-09-11T05:16:13.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7528070010 msgr2=0x7f75281d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7527fff640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7528070010 0x7f75281d01e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f751800c480 tx=0x7f751800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 client.?.objecter ms_handle_connect 0x7f7528070010 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518012a40 con 0x7f7528070010 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: dump: 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.967 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: _finish_auth 0 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7518017070 con 0x7f7528070010 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 5 monclient: authenticate success, global_id 64455 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient: _renew_subs 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75280aac50 con 0x7f7528070010 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518012e40 con 0x7f7528070010 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:13.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 10 monclient: dump: 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f75267fc640 4 set_mon_vals no callback set 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 client.64455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient: _renew_subs 2024-09-11T05:16:13.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75281362d0 con 0x7f7528070010 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 librados: init done 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752813f7d0 0x7f75280a3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7528137860 con 0x7f752813f7d0 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f75277fe640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752813f7d0 0x7f75280a3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f75277fe640 10 monclient: get_auth_request con 0x7f752813f7d0 auth_method 0 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f75277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.949+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f751801c460 con 0x7f7528070010 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 10 monclient(hunting): handle_auth_done global_id 84131 payload 824 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 1 -- 172.21.15.138:0/1630668638 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7528136500 con 0x7f752813f7d0 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 1 --2- 172.21.15.138:0/1630668638 >> v2:172.21.15.190:6800/1562782189 conn(0x7f751403ced0 0x7f751403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (secure 0 0 0) 0x7f751804f7f0 con 0x7f7528070010 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 client.64455.objecter ms_dispatch 0x7f7528000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 3 client.64455.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 3 client.64455.objecter handle_osd_map decoding full epoch 365 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 20 client.64455.objecter dump_active .. 0 homeless 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f752cde8640 1 --2- 172.21.15.138:0/1630668638 >> v2:172.21.15.190:6800/1562782189 conn(0x7f751403ced0 0x7f751403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f752cde8640 10 monclient: get_auth_request con 0x7f751403ced0 auth_method 0 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 client.64455.objecter ms_handle_connect 0x7f752813f7d0 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752813f7d0 msgr2=0x7f75280a3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 client.64455.objecter resend_mon_ops 2024-09-11T05:16:13.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f75277fe640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752813f7d0 0x7f75280a3ad0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f751c0ad000 tx=0x7f751c019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0ba020 con 0x7f752813f7d0 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 monclient: dump: 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.972 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f751c009490 con 0x7f752813f7d0 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752813f7d0 msgr2=0x7f75280a3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752813f7d0 0x7f75280a3ad0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f751c0ad000 tx=0x7f751c019150 comp rx=0 tx=0).stop 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752813f7d0 msgr2=0x7f75280a3ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f752813f7d0 0x7f75280a3ad0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f7524ff9640 10 client.64455.objecter ms_handle_connect 0x7f751403ced0 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.953+0000 7f752cde8640 1 --2- 172.21.15.138:0/1630668638 >> v2:172.21.15.190:6800/1562782189 conn(0x7f751403ced0 0x7f751403f390 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7f7528137fc0 tx=0x7f7520007780 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f752e84b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f752e84b640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75280a4010 0x7f752809f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f752e84b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f75280a4420 con 0x7f75280a4010 2024-09-11T05:16:13.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75280a4010 0x7f752809f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 10 monclient: get_auth_request con 0x7f75280a4010 auth_method 0 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 10 monclient(hunting): handle_auth_done global_id 84134 payload 824 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 1 -- 172.21.15.138:0/1630668638 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7528135450 con 0x7f75280a4010 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75280a4010 msgr2=0x7f752809f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f75277fe640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75280a4010 0x7f752809f030 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f751c000c10 tx=0x7f751c0a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 10 client.64455.objecter ms_handle_connect 0x7f75280a4010 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 10 client.64455.objecter resend_mon_ops 2024-09-11T05:16:13.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0c8070 con 0x7f75280a4010 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 10 monclient: dump: 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:13.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f751c0c1860 con 0x7f75280a4010 2024-09-11T05:16:13.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:16:13.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:13.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75280a4010 msgr2=0x7f752809f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75280a4010 0x7f752809f030 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f751c000c10 tx=0x7f751c0a8000 comp rx=0 tx=0).stop 2024-09-11T05:16:13.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75280a4010 msgr2=0x7f752809f030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:13.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.961+0000 7f7524ff9640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f75280a4010 0x7f752809f030 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f751c000c10 tx=0x7f751c0a8000 comp rx=0 tx=0).stop 2024-09-11T05:16:13.976 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 32, 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 17731, 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 7, 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:16:13.977 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:16:13.978 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:16:13.979 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:16:13.980 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:13.981 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.982 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:13.983 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:16:13.985 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:13.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:13.987 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:13.988 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:13.988 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:13.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:13.988 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:16:13.988 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 10 librados: watch_flush enter 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 10 librados: watch_flush exit 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 20 client.64455.objecter shutdown clearing up homeless session... 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 10 client.64455.objecter successfully canceled tick 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 >> v2:172.21.15.190:6800/1562782189 conn(0x7f751403ced0 msgr2=0x7f751403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 --2- 172.21.15.138:0/1630668638 >> v2:172.21.15.190:6800/1562782189 conn(0x7f751403ced0 0x7f751403f390 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7f7528137fc0 tx=0x7f7520007780 comp rx=0 tx=0).stop 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 10 monclient: shutdown 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7528070010 msgr2=0x7f75281d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 --2- 172.21.15.138:0/1630668638 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7528070010 0x7f75281d01e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f751800c480 tx=0x7f751800c940 comp rx=0 tx=0).stop 2024-09-11T05:16:13.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752cde8640 1 -- 172.21.15.138:0/1630668638 reap_dead start 2024-09-11T05:16:13.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752cde8640 1 -- 172.21.15.138:0/1630668638 reap_dead start 2024-09-11T05:16:13.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 shutdown_connections 2024-09-11T05:16:13.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 >> 172.21.15.138:0/1630668638 conn(0x7f75280ab830 msgr2=0x7f752806ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:13.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 shutdown_connections 2024-09-11T05:16:13.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 -- 172.21.15.138:0/1630668638 wait complete. 2024-09-11T05:16:13.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:13.965+0000 7f752e84b640 1 librados: shutdown 2024-09-11T05:16:13.991 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:16:13.991 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:16:13.991 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.089+0000 7f108cdba640 10 monclient: get_monmap_and_config 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.089+0000 7f108cdba640 10 monclient: build_initial_monmap 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.089+0000 7f108cdba640 10 monclient: monmap: 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:14.096265+0000 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:14.096265+0000 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:14.093 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:14.094 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:14.094 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:14.094 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:14.094 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:14.094 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.089+0000 7f108cdba640 1 Processor -- start 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.089+0000 7f108cdba640 1 -- start start 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.089+0000 7f108cdba640 10 monclient: init 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1088138a90 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f108806d440 0x7f108813d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient(hunting): picked mon.noname-b con 0x7f108806d440 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f108813dd30 0x7f1088140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient(hunting): picked mon.noname-c con 0x7f108813dd30 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:14.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f108806d1c0 con 0x7f1088138a90 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f108809ce60 con 0x7f108813dd30 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f108809cb80 con 0x7f108806d440 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f108cdba640 10 monclient(hunting): authenticate will time out at 1457.532318s 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f108806d440 0x7f108813d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f108806d440 0x7f108813d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:56982/0 (socket says 172.21.15.138:56982) 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 1 -- 172.21.15.138:0/3580217126 learned_addr learned my addr 172.21.15.138:0/3580217126 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient(hunting): get_auth_request con 0x7f108806d440 auth_method 0 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:14.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1087fff640 1 --2- 172.21.15.138:0/3580217126 >> v2:172.21.15.190:3300/0 conn(0x7f108813dd30 0x7f1088140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1087fff640 10 monclient(hunting): get_auth_request con 0x7f108813dd30 auth_method 0 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1087fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1087fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1087fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient(hunting): handle_auth_done global_id 84137 payload 274 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 1 -- 172.21.15.138:0/3580217126 >> v2:172.21.15.190:3300/0 conn(0x7f108813dd30 msgr2=0x7f1088140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 1 --2- 172.21.15.138:0/3580217126 >> v2:172.21.15.190:3300/0 conn(0x7f108813dd30 0x7f1088140120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 1 -- 172.21.15.138:0/3580217126 >> v1:172.21.15.190:6789/0 conn(0x7f1088138a90 legacy=0x7f1088138e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 1 monclient: found mon.noname-b 2024-09-11T05:16:14.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:14.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:16:14.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.093+0000 7f1086ffd640 1 -- 172.21.15.138:0/3580217126 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108809d140 con 0x7f108806d440 2024-09-11T05:16:14.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 -- 172.21.15.138:0/3580217126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f108806d440 msgr2=0x7f108813d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:14.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 --2- 172.21.15.138:0/3580217126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f108806d440 0x7f108813d7f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f107c009870 tx=0x7f107c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8809d3645a371c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:14.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 1 -- 172.21.15.138:0/3580217126 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c0b1070 con 0x7f108806d440 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: dump: 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:14.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: _finish_auth 0 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: _check_auth_tickets 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 5 monclient: authenticate success, global_id 84137 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 1 -- 172.21.15.138:0/3580217126 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107c0ac620 con 0x7f108806d440 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 1 -- 172.21.15.138:0/3580217126 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c0ac9f0 con 0x7f108806d440 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 4 set_mon_vals no callback set 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10867fc640 10 monclient: dump: 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:14.104 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient: shutdown 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- 172.21.15.138:0/3580217126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f108806d440 msgr2=0x7f108813d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 --2- 172.21.15.138:0/3580217126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f108806d440 0x7f108813d7f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f107c009870 tx=0x7f107c01c420 comp rx=0 tx=0).stop 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- 172.21.15.138:0/3580217126 shutdown_connections 2024-09-11T05:16:14.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 --2- 172.21.15.138:0/3580217126 >> v2:172.21.15.190:3300/0 conn(0x7f108813dd30 0x7f1088140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 --2- 172.21.15.138:0/3580217126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f108806d440 0x7f108813d7f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- 172.21.15.138:0/3580217126 >> 172.21.15.138:0/3580217126 conn(0x7f10880a48c0 msgr2=0x7f10880a4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- 172.21.15.138:0/3580217126 shutdown_connections 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- 172.21.15.138:0/3580217126 wait complete. 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient: build_initial_monmap 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient: monmap: 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:14.106 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 librados: starting msgr at 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 librados: starting objecter 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 Processor -- start 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- start start 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 librados: setting wanted keys 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 librados: calling monclient init 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient: init 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1088138a90 0x7f108811ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient(hunting): picked mon.noname-b con 0x7f1088138a90 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f108813dd30 0x7f1088135890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:14.107 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient(hunting): picked mon.noname-a con 0x7f108813dd30 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1088135dd0 0x7f10881d0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient(hunting): picked mon.noname-c con 0x7f1088135dd0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1088143260 con 0x7f108813dd30 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f10881430e0 con 0x7f1088138a90 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f10881433e0 con 0x7f1088135dd0 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f108cdba640 10 monclient(hunting): authenticate will time out at 1457.539273s 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1088135dd0 0x7f10881d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f108813dd30 0x7f1088135890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1088135dd0 0x7f10881d0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:56994/0 (socket says 172.21.15.138:56994) 2024-09-11T05:16:14.108 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 1 -- 172.21.15.138:0/2494408166 learned_addr learned my addr 172.21.15.138:0/2494408166 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 10 monclient(hunting): get_auth_request con 0x7f1088135dd0 auth_method 0 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient(hunting): get_auth_request con 0x7f108813dd30 auth_method 0 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:14.109 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10877fe640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1088138a90 0x7f108811ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient(hunting): handle_auth_done global_id 64458 payload 824 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1088135dd0 msgr2=0x7f10881d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1088135dd0 0x7f10881d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1088138a90 msgr2=0x7f108811ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1088138a90 0x7f108811ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 monclient: found mon.noname-a 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1086ffd640 1 -- 172.21.15.138:0/2494408166 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108809f0b0 con 0x7f108813dd30 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10877fe640 10 monclient: get_auth_request con 0x7f1088138a90 auth_method 0 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f1087fff640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1088135dd0 0x7f10881d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:16:14.110 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.097+0000 7f10877fe640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1088138a90 0x7f108811ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f1086ffd640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f108813dd30 msgr2=0x7f1088135890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f1086ffd640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f108813dd30 0x7f1088135890 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f107c0ac000 tx=0x7f107c01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 client.?.objecter ms_handle_connect 0x7f108813dd30 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c0b1040 con 0x7f108813dd30 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: dump: 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.114 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: _finish_auth 0 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107c0bc070 con 0x7f108813dd30 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 5 monclient: authenticate success, global_id 64458 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 10 monclient: _renew_subs 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:14.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108813d1d0 con 0x7f108813dd30 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c0b7690 con 0x7f108813dd30 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 monclient: dump: 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:14.116 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f1085ffb640 4 set_mon_vals no callback set 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 10 client.64458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 10 monclient: _renew_subs 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f108811f980 con 0x7f108813dd30 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 1 librados: init done 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108806d440 0x7f10881d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1088138e70 con 0x7f108806d440 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f107c0b7830 con 0x7f108813dd30 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108806d440 0x7f10881d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 10 monclient: get_auth_request con 0x7f108806d440 auth_method 0 2024-09-11T05:16:14.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 1 --2- 172.21.15.138:0/2494408166 >> v2:172.21.15.190:6800/1562782189 conn(0x7f107403ced0 0x7f107403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 5390+0+0 (secure 0 0 0) 0x7f107c0e9c70 con 0x7f108813dd30 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 10 client.64458.objecter ms_dispatch 0x7f1088000ba0 osd_map(365..365 src has 1..365) 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 3 client.64458.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 3 client.64458.objecter handle_osd_map decoding full epoch 365 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f106ffff640 20 client.64458.objecter dump_active .. 0 homeless 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 10 monclient(hunting): handle_auth_done global_id 74185 payload 824 2024-09-11T05:16:14.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f1087fff640 1 --2- 172.21.15.138:0/2494408166 >> v2:172.21.15.190:6800/1562782189 conn(0x7f107403ced0 0x7f107403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f10877fe640 1 -- 172.21.15.138:0/2494408166 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f108806cdf0 con 0x7f108806d440 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.101+0000 7f1087fff640 10 monclient: get_auth_request con 0x7f107403ced0 auth_method 0 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 10 client.64458.objecter ms_handle_connect 0x7f108806d440 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 10 client.64458.objecter resend_mon_ops 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f10877fe640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108806d440 msgr2=0x7f10881d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f10877fe640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108806d440 0x7f10881d0d60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f107800d480 tx=0x7f107800d940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078012a10 con 0x7f108806d440 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 10 monclient: dump: 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.119 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 10 client.64458.objecter ms_handle_connect 0x7f107403ced0 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f1087fff640 1 --2- 172.21.15.138:0/2494408166 >> v2:172.21.15.190:6800/1562782189 conn(0x7f107403ced0 0x7f107403f390 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f10800094c0 tx=0x7f10800098b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1078012a10 con 0x7f108806d440 2024-09-11T05:16:14.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:14.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:14.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108806d440 msgr2=0x7f10881d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:14.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108806d440 0x7f10881d0d60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f107800d480 tx=0x7f107800d940 comp rx=0 tx=0).stop 2024-09-11T05:16:14.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108806d440 msgr2=0x7f10881d0d60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:14.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.105+0000 7f106ffff640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108806d440 0x7f10881d0d60 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f107800d480 tx=0x7f107800d940 comp rx=0 tx=0).stop 2024-09-11T05:16:14.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f108cdba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:16:14.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f108cdba640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108811fb50 0x7f10881d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f108cdba640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f10880a6f10 con 0x7f108811fb50 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108811fb50 0x7f10881d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 10 monclient: get_auth_request con 0x7f108811fb50 auth_method 0 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 10 monclient(hunting): handle_auth_done global_id 74188 payload 824 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.113+0000 7f10877fe640 1 -- 172.21.15.138:0/2494408166 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f108806dd00 con 0x7f108811fb50 2024-09-11T05:16:14.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f10877fe640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108811fb50 msgr2=0x7f10881d12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 10 client.64458.objecter ms_handle_connect 0x7f108811fb50 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f10877fe640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108811fb50 0x7f10881d12a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1078017000 tx=0x7f1078003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 10 client.64458.objecter resend_mon_ops 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078028020 con 0x7f108811fb50 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 10 monclient: dump: 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:14.124 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f1078023350 con 0x7f108811fb50 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108811fb50 msgr2=0x7f10881d12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108811fb50 0x7f10881d12a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1078017000 tx=0x7f1078003770 comp rx=0 tx=0).stop 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108811fb50 msgr2=0x7f10881d12a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f106ffff640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f108811fb50 0x7f10881d12a0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1078017000 tx=0x7f1078003770 comp rx=0 tx=0).stop 2024-09-11T05:16:14.125 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 32, 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 141670, 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 7, 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:16:14.126 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:16:14.127 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:16:14.128 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:16:14.129 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:14.130 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.132 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:14.133 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.134 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:14.135 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:16:14.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:16:14.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:14.138 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:14.138 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:14.138 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:14.138 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:14.138 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:14.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:14.138 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:16:14.138 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 10 librados: watch_flush enter 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 10 librados: watch_flush exit 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 20 client.64458.objecter shutdown clearing up homeless session... 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 10 client.64458.objecter successfully canceled tick 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 >> v2:172.21.15.190:6800/1562782189 conn(0x7f107403ced0 msgr2=0x7f107403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 --2- 172.21.15.138:0/2494408166 >> v2:172.21.15.190:6800/1562782189 conn(0x7f107403ced0 0x7f107403f390 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f10800094c0 tx=0x7f10800098b0 comp rx=0 tx=0).stop 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 10 monclient: shutdown 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:14.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f108813dd30 msgr2=0x7f1088135890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:14.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f108813dd30 0x7f1088135890 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f107c0ac000 tx=0x7f107c01ac60 comp rx=0 tx=0).stop 2024-09-11T05:16:14.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f1087fff640 1 -- 172.21.15.138:0/2494408166 reap_dead start 2024-09-11T05:16:14.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 shutdown_connections 2024-09-11T05:16:14.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 --2- 172.21.15.138:0/2494408166 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f108813dd30 0x7f1088135890 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:14.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 >> 172.21.15.138:0/2494408166 conn(0x7f10880a48c0 msgr2=0x7f10880a3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:14.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 shutdown_connections 2024-09-11T05:16:14.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 -- 172.21.15.138:0/2494408166 wait complete. 2024-09-11T05:16:14.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:14.117+0000 7f108cdba640 1 librados: shutdown 2024-09-11T05:16:14.141 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:16:14.142 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T05:16:14.142 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-11T05:16:14.142 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T05:16:14.142 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T05:16:14.142 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T05:16:14.142 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:16:14.169 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T05:16:14.169 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-11T05:16:14.170 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-11T05:16:14.170 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T05:16:29.171 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-11T05:16:29.172 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-11T05:16:29.172 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:16:29.172 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:16:29.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient: get_monmap_and_config 2024-09-11T05:16:29.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient: build_initial_monmap 2024-09-11T05:16:29.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient: monmap: 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:29.303036+0000 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:29.303036+0000 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:29.300 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 1 Processor -- start 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 1 -- start start 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient: init 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 0x7f0f480a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient(hunting): picked mon.noname-c con 0x7f0f480a0a90 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient(hunting): picked mon.noname-b con 0x7f0f4809dc20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f480a2bb0 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f4809efa0 con 0x7f0f480a2bb0 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:29.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0f480a37b0 con 0x7f0f480a0a90 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4809ecc0 con 0x7f0f4809dc20 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f4da01640 10 monclient(hunting): authenticate will time out at 1472.739661s 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f46ffd640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 msgr2=0x7f0f480a0e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f46ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 0x7f0f480a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59624/0 (socket says 172.21.15.138:59624) 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 learned_addr learned my addr 172.21.15.138:0/3127061718 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f477fe640 1 --1- 172.21.15.138:0/3127061718 >> v1:172.21.15.138:6789/0 conn(0x7f0f480a2bb0 0x7f0f481348c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:40578/0 (socket says 172.21.15.138:40578) 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f4809dc20 auth_method 0 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:29.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:29.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:16:29.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:29.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:29.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.297+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:29.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.301+0000 7f0f477fe640 1 -- 172.21.15.138:0/3127061718 >> v1:172.21.15.138:6789/0 conn(0x7f0f480a2bb0 legacy=0x7f0f481348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:29.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f46ffd640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 msgr2=0x7f0f480a0e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:16:29.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f46ffd640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 0x7f0f480a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f4809dc20 auth_method 0 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:29.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.501+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:16:29.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f46ffd640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 msgr2=0x7f0f480a0e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:16:29.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f46ffd640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 0x7f0f480a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f4809dc20 auth_method 0 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:29.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:29.905+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:16:30.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f46ffd640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 msgr2=0x7f0f480a0e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:16:30.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f46ffd640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 0x7f0f480a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:16:30.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:30.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f4809dc20 auth_method 0 2024-09-11T05:16:30.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:30.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:30.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:30.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:16:30.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:30.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:30.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:30.709+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:16:32.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 10 monclient(hunting): tick 2024-09-11T05:16:32.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:16:32.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 monclient(hunting): continuing hunt 2024-09-11T05:16:32.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:16:32.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 msgr2=0x7f0f480a2670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:32.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f4809dc20 0x7f0f480a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:32.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 msgr2=0x7f0f480a0e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:32.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.190:3300/0 conn(0x7f0f480a0a90 0x7f0f480a0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:32.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 >> v1:172.21.15.138:6789/0 conn(0x7f0f480a2bb0 legacy=0x7f0f481348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:32.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 10 monclient: _add_conns ranks=[1,3] 2024-09-11T05:16:32.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f34003770 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:32.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:32.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0f34007400 addr v2:172.21.15.138:3300/0 2024-09-11T05:16:32.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f480a13b0 con 0x7f0f34003770 2024-09-11T05:16:32.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:32.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f0f480a37b0 con 0x7f0f34007400 2024-09-11T05:16:32.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f44ff9640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:32.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f46ffd640 1 -- 172.21.15.138:0/3127061718 >> v1:172.21.15.190:6789/0 conn(0x7f0f34003770 legacy=0x7f0f34005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:16:32.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:32.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f34007400 auth_method 0 2024-09-11T05:16:32.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:32.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:32.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:32.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:16:32.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:32.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:32.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.301+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:32.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f46ffd640 1 -- 172.21.15.138:0/3127061718 >> v1:172.21.15.190:6789/0 conn(0x7f0f34003770 legacy=0x7f0f34005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:16:32.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:32.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f34007400 auth_method 0 2024-09-11T05:16:32.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:32.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:32.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:32.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:16:32.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:32.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:32.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.505+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:16:32.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:32.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f46ffd640 1 -- 172.21.15.138:0/3127061718 >> v1:172.21.15.190:6789/0 conn(0x7f0f34003770 legacy=0x7f0f34005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:16:32.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f34007400 auth_method 0 2024-09-11T05:16:32.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:32.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:32.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:32.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:16:32.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:32.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:32.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:32.909+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:16:33.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f46ffd640 1 -- 172.21.15.138:0/3127061718 >> v1:172.21.15.190:6789/0 conn(0x7f0f34003770 legacy=0x7f0f34005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:16:33.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:33.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f34007400 auth_method 0 2024-09-11T05:16:33.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:33.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:33.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:16:33.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:16:33.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:16:33.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:16:33.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:33.713+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): tick 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 monclient(hunting): continuing hunt 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 >> v1:172.21.15.190:6789/0 conn(0x7f0f34003770 legacy=0x7f0f34005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f477fe640 1 -- 172.21.15.138:0/3127061718 reap_dead start 2024-09-11T05:16:35.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0f34007400 addr v2:172.21.15.138:3300/0 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34006300 0x7f0f34005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0f34006300 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f34006700 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3400aff0 con 0x7f0f34006700 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f0f480a37b0 con 0x7f0f34007400 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 1 -- 172.21.15.138:0/3127061718 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4809ecc0 con 0x7f0f34006300 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f44ff9640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f46ffd640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34006300 0x7f0f34005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f34006300 auth_method 0 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f46ffd640 10 monclient(hunting): get_auth_request con 0x7f0f34007400 auth_method 0 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient(hunting): handle_auth_done global_id 84143 payload 274 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 msgr2=0x7f0f340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> v1:172.21.15.138:6789/0 conn(0x7f0f34006700 legacy=0x7f0f34006bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient: _finish_hunting 0 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 1 monclient: found mon.noname-b 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:16:35.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.301+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f480a13b0 con 0x7f0f34006300 2024-09-11T05:16:35.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f467fc640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34006300 msgr2=0x7f0f34005b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f467fc640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34006300 0x7f0f34005b60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00b310 tx=0x7f0f3c020290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f9e92a248dfec8f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 1 -- 172.21.15.138:0/3127061718 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0b4070 con 0x7f0f34006300 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: dump: 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: _finish_auth 0 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 1 -- 172.21.15.138:0/3127061718 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f3c004440 con 0x7f0f34006300 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 1 -- 172.21.15.138:0/3127061718 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c003860 con 0x7f0f34006300 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f45ffb640 10 monclient: dump: 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f4da01640 5 monclient: authenticate success, global_id 84143 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f4da01640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:35.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.305+0000 7f0f4da01640 4 set_mon_vals no callback set 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient: shutdown 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34006300 msgr2=0x7f0f34005b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34006300 0x7f0f34005b60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00b310 tx=0x7f0f3c020290 comp rx=0 tx=0).stop 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- 172.21.15.138:0/3127061718 shutdown_connections 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 --2- 172.21.15.138:0/3127061718 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34006300 0x7f0f34005b60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 --2- 172.21.15.138:0/3127061718 >> v2:172.21.15.138:3300/0 conn(0x7f0f34007400 0x7f0f340098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- 172.21.15.138:0/3127061718 >> 172.21.15.138:0/3127061718 conn(0x7f0f48063640 msgr2=0x7f0f48063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- 172.21.15.138:0/3127061718 shutdown_connections 2024-09-11T05:16:35.315 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- 172.21.15.138:0/3127061718 wait complete. 2024-09-11T05:16:35.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient: build_initial_monmap 2024-09-11T05:16:35.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient: monmap: 2024-09-11T05:16:35.316 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:35.316 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:35.316 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:35.316 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 librados: starting msgr at 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 librados: starting objecter 2024-09-11T05:16:35.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 Processor -- start 2024-09-11T05:16:35.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- start start 2024-09-11T05:16:35.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 librados: setting wanted keys 2024-09-11T05:16:35.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 librados: calling monclient init 2024-09-11T05:16:35.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient: init 2024-09-11T05:16:35.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f34006700 0x7f0f481a9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient(hunting): picked mon.noname-a con 0x7f0f34006700 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34007400 0x7f0f481c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient(hunting): picked mon.noname-c con 0x7f0f34007400 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f481c30a0 0x7f0f481c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient(hunting): picked mon.noname-b con 0x7f0f481c30a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0f480a2a00 con 0x7f0f34006700 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f480a2880 con 0x7f0f481c30a0 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4806e480 con 0x7f0f34007400 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.309+0000 7f0f4da01640 10 monclient(hunting): authenticate will time out at 1478.751877s 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f34006700 0x7f0f481a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f477fe640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f481c30a0 msgr2=0x7f0f481c5ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f477fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f481c30a0 0x7f0f481c5ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f34006700 0x7f0f481a9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48798/0 (socket says 172.21.15.138:48798) 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 1 -- 172.21.15.138:0/2952080549 learned_addr learned my addr 172.21.15.138:0/2952080549 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 10 monclient(hunting): get_auth_request con 0x7f0f34006700 auth_method 0 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 --2- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34007400 0x7f0f481c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient(hunting): get_auth_request con 0x7f0f34007400 auth_method 0 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient(hunting): handle_auth_done global_id 84146 payload 824 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 -- 172.21.15.138:0/2952080549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f481c30a0 msgr2=0x7f0f481c5ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:16:35.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 --2- 172.21.15.138:0/2952080549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f481c30a0 0x7f0f481c5ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 -- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f34006700 msgr2=0x7f0f481a9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 --2- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f34006700 0x7f0f481a9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient: _finish_hunting 0 2024-09-11T05:16:35.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 monclient: found mon.noname-c 2024-09-11T05:16:35.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:35.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:16:35.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 -- 172.21.15.138:0/2952080549 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f481a9bb0 con 0x7f0f34007400 2024-09-11T05:16:35.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 -- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34007400 msgr2=0x7f0f481c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f467fc640 1 --2- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34007400 0x7f0f481c95c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400e550 tx=0x7f0f3c01eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 client.?.objecter ms_handle_connect 0x7f0f34007400 2024-09-11T05:16:35.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:35.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 1 -- 172.21.15.138:0/2952080549 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0bd070 con 0x7f0f34007400 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: dump: 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: _finish_auth 0 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 1 -- 172.21.15.138:0/2952080549 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f3c003ae0 con 0x7f0f34007400 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 1 -- 172.21.15.138:0/2952080549 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0b4040 con 0x7f0f34007400 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f2ffff640 10 monclient: dump: 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.326 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 5 monclient: authenticate success, global_id 84146 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 10 monclient: _renew_subs 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f457fa640 4 set_mon_vals no callback set 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f4809e110 con 0x7f0f34007400 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 10 client.84146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 10 monclient: _renew_subs 2024-09-11T05:16:35.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f481aaa20 con 0x7f0f34007400 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.313+0000 7f0f4da01640 1 librados: init done 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f2ffff640 1 -- 172.21.15.138:0/2952080549 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f0f3c0b7070 con 0x7f0f34007400 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f4da01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f4da01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f4da01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f480a10c0 con 0x7f0f34007400 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f2ffff640 1 --2- 172.21.15.138:0/2952080549 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0f34040f20 0x7f0f340433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f2ffff640 1 -- 172.21.15.138:0/2952080549 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(366..366 src has 1..366) ==== 5005+0+0 (secure 0 0 0) 0x7f0f3c0ea6f0 con 0x7f0f34007400 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f2ffff640 10 client.84146.objecter ms_dispatch 0x7f0f48000ba0 osd_map(366..366 src has 1..366) 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f2ffff640 3 client.84146.objecter handle_osd_map got epochs [366,366] > 0 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f2ffff640 3 client.84146.objecter handle_osd_map decoding full epoch 366 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f2ffff640 20 client.84146.objecter dump_active .. 0 homeless 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f46ffd640 1 --2- 172.21.15.138:0/2952080549 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0f34040f20 0x7f0f340433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.317+0000 7f0f46ffd640 10 monclient: get_auth_request con 0x7f0f34040f20 auth_method 0 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.321+0000 7f0f2ffff640 10 client.84146.objecter ms_handle_connect 0x7f0f34040f20 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.321+0000 7f0f46ffd640 1 --2- 172.21.15.138:0/2952080549 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0f34040f20 0x7f0f340433e0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7f0f38003e10 tx=0x7f0f380075c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.321+0000 7f0f2ffff640 1 -- 172.21.15.138:0/2952080549 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f0f3c018040 con 0x7f0f34007400 2024-09-11T05:16:35.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.321+0000 7f0f2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:35.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.321+0000 7f0f2ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:35.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.565+0000 7f0f4da01640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:16:35.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.565+0000 7f0f4da01640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:16:35.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.565+0000 7f0f4da01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:16:35.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.565+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0f481aa180 con 0x7f0f34007400 2024-09-11T05:16:35.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.565+0000 7f0f2ffff640 1 -- 172.21.15.138:0/2952080549 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f0f3c0e9010 con 0x7f0f34007400 2024-09-11T05:16:35.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.565+0000 7f0f2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:16:35.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.565+0000 7f0f2ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:35.569 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":36,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 10 librados: watch_flush enter 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 10 librados: watch_flush exit 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 20 client.84146.objecter shutdown clearing up homeless session... 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 10 client.84146.objecter successfully canceled tick 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0f34040f20 msgr2=0x7f0f340433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 --2- 172.21.15.138:0/2952080549 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0f34040f20 0x7f0f340433e0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7f0f38003e10 tx=0x7f0f380075c0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 10 monclient: shutdown 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34007400 msgr2=0x7f0f481c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 --2- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34007400 0x7f0f481c95c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400e550 tx=0x7f0f3c01eb30 comp rx=0 tx=0).stop 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 shutdown_connections 2024-09-11T05:16:35.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 --2- 172.21.15.138:0/2952080549 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0f34040f20 0x7f0f340433e0 unknown :-1 s=CLOSED pgs=3427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 --2- 172.21.15.138:0/2952080549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f481c30a0 0x7f0f481c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 --2- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0f34007400 0x7f0f481c95c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 --2- 172.21.15.138:0/2952080549 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0f34006700 0x7f0f481a9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 >> 172.21.15.138:0/2952080549 conn(0x7f0f48063640 msgr2=0x7f0f48073ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:35.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 shutdown_connections 2024-09-11T05:16:35.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 -- 172.21.15.138:0/2952080549 wait complete. 2024-09-11T05:16:35.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.569+0000 7f0f4da01640 1 librados: shutdown 2024-09-11T05:16:35.587 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:16:35.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T05:16:35.588 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:16:35.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient: get_monmap_and_config 2024-09-11T05:16:35.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient: build_initial_monmap 2024-09-11T05:16:35.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient: monmap: 2024-09-11T05:16:35.713 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:35.713 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.713 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:35.715808+0000 2024-09-11T05:16:35.713 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:35.715808+0000 2024-09-11T05:16:35.713 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:35.714 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.714 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:35.714 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:35.714 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:35.714 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:35.714 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:35.714 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 1 Processor -- start 2024-09-11T05:16:35.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 1 -- start start 2024-09-11T05:16:35.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient: init 2024-09-11T05:16:35.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:35.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-11T05:16:35.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f326409da20 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:35.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f32640a2030 0x7f326413b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient(hunting): picked mon.noname-a con 0x7f32640a2030 addr v2:172.21.15.138:3300/0 2024-09-11T05:16:35.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32640a24a0 addr v1:172.21.15.190:6789/0 2024-09-11T05:16:35.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3264064590 con 0x7f326409da20 2024-09-11T05:16:35.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32640a2c70 con 0x7f32640a24a0 2024-09-11T05:16:35.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f326409de00 con 0x7f32640a2030 2024-09-11T05:16:35.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.709+0000 7f326a4b4640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:35.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f326a4b4640 10 monclient(hunting): authenticate will time out at 1479.152014s 2024-09-11T05:16:35.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f32640a2030 0x7f326413b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f32640a2030 0x7f326413b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48810/0 (socket says 172.21.15.138:48810) 2024-09-11T05:16:35.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 1 -- 172.21.15.138:0/3669559418 learned_addr learned my addr 172.21.15.138:0/3669559418 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:35.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient(hunting): get_auth_request con 0x7f32640a2030 auth_method 0 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f3263fff640 1 --1- 172.21.15.138:0/3669559418 >> v1:172.21.15.138:6789/0 conn(0x7f326409da20 0x7f32640a1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:40604/0 (socket says 172.21.15.138:40604) 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f3268a51640 1 -- 172.21.15.138:0/3669559418 >> v1:172.21.15.190:6789/0 conn(0x7f32640a24a0 legacy=0x7f326413da70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient(hunting): handle_auth_done global_id 64467 payload 274 2024-09-11T05:16:35.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 1 -- 172.21.15.138:0/3669559418 >> v1:172.21.15.190:6789/0 conn(0x7f32640a24a0 legacy=0x7f326413da70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:35.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 1 -- 172.21.15.138:0/3669559418 >> v1:172.21.15.138:6789/0 conn(0x7f326409da20 legacy=0x7f32640a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:35.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient: _finish_hunting 0 2024-09-11T05:16:35.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 1 monclient: found mon.noname-a 2024-09-11T05:16:35.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:35.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:35.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.713+0000 7f32637fe640 1 -- 172.21.15.138:0/3669559418 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32640a2f50 con 0x7f32640a2030 2024-09-11T05:16:35.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f32637fe640 1 -- 172.21.15.138:0/3669559418 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32640a2030 msgr2=0x7f326413b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f32637fe640 1 --2- 172.21.15.138:0/3669559418 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32640a2030 0x7f326413b520 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f324c009870 tx=0x7f324c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788cd269686292e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 1 -- 172.21.15.138:0/3669559418 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c0b1070 con 0x7f32640a2030 2024-09-11T05:16:35.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:35.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: dump: 2024-09-11T05:16:35.734 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.734 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: _finish_auth 0 2024-09-11T05:16:35.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 1 -- 172.21.15.138:0/3669559418 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f324c0ac620 con 0x7f32640a2030 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 1 -- 172.21.15.138:0/3669559418 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c0ac9f0 con 0x7f32640a2030 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 5 monclient: authenticate success, global_id 64467 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 4 set_mon_vals no callback set 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f3262ffd640 10 monclient: dump: 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.736 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.737 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.737 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.737 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.737 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.737 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.737 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 10 monclient: shutdown 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 -- 172.21.15.138:0/3669559418 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32640a2030 msgr2=0x7f326413b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 --2- 172.21.15.138:0/3669559418 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32640a2030 0x7f326413b520 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f324c009870 tx=0x7f324c01c420 comp rx=0 tx=0).stop 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 -- 172.21.15.138:0/3669559418 shutdown_connections 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 --2- 172.21.15.138:0/3669559418 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f32640a2030 0x7f326413b520 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 -- 172.21.15.138:0/3669559418 >> 172.21.15.138:0/3669559418 conn(0x7f3264063640 msgr2=0x7f3264063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 -- 172.21.15.138:0/3669559418 shutdown_connections 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 -- 172.21.15.138:0/3669559418 wait complete. 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 10 monclient: build_initial_monmap 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 10 monclient: monmap: 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:35.738 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 librados: starting msgr at 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 librados: starting objecter 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 Processor -- start 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 -- start start 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 librados: setting wanted keys 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 1 librados: calling monclient init 2024-09-11T05:16:35.739 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.717+0000 7f326a4b4640 10 monclient: init 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f326409da20 0x7f326413dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient(hunting): picked mon.noname-a con 0x7f326409da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f32640a2030 0x7f326413cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient(hunting): picked mon.noname-c con 0x7f32640a2030 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32640a24a0 0x7f3264070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient(hunting): picked mon.noname-b con 0x7f32640a24a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3264141d40 con 0x7f326409da20 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3264141bc0 con 0x7f32640a24a0 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3264141ec0 con 0x7f32640a2030 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient(hunting): authenticate will time out at 1479.160306s 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f326409da20 0x7f326413dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f32640a2030 0x7f326413cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f326409da20 0x7f326413dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48816/0 (socket says 172.21.15.138:48816) 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 -- 172.21.15.138:0/2854622802 learned_addr learned my addr 172.21.15.138:0/2854622802 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient(hunting): get_auth_request con 0x7f326409da20 auth_method 0 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 10 monclient(hunting): get_auth_request con 0x7f32640a2030 auth_method 0 2024-09-11T05:16:35.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3268a51640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32640a24a0 msgr2=0x7f3264070a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3268a51640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32640a24a0 0x7f3264070a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient(hunting): handle_auth_done global_id 64470 payload 824 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f32640a2030 msgr2=0x7f326413cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f32640a2030 0x7f326413cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32640a24a0 msgr2=0x7f3264070a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32640a24a0 0x7f3264070a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient: _finish_hunting 0 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 monclient: found mon.noname-a 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 -- 172.21.15.138:0/2854622802 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326412df30 con 0x7f326409da20 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32637fe640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f32640a2030 0x7f326413cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f326409da20 msgr2=0x7f326413dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3263fff640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f326409da20 0x7f326413dcd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f325c00c820 tx=0x7f325c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 client.?.objecter ms_handle_connect 0x7f326409da20 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c017070 con 0x7f326409da20 2024-09-11T05:16:35.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: dump: 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: _finish_auth 0 2024-09-11T05:16:35.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325c00a860 con 0x7f326409da20 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 5 monclient: authenticate success, global_id 64470 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient: _renew_subs 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326409e110 con 0x7f326409da20 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c00ac50 con 0x7f326409da20 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f3260ff9640 10 monclient: dump: 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.745 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f32627fc640 4 set_mon_vals no callback set 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 client.64470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient: _renew_subs 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:35.746 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32640742b0 con 0x7f326409da20 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 1 librados: init done 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.721+0000 7f326a4b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f326a4b4640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264070f90 0x7f3264071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f326a4b4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3264130370 con 0x7f3264070f90 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264070f90 0x7f3264071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 10 monclient: get_auth_request con 0x7f3264070f90 auth_method 0 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 10 monclient(hunting): handle_auth_done global_id 64473 payload 824 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 1 -- 172.21.15.138:0/2854622802 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32640718b0 con 0x7f3264070f90 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f325c007070 con 0x7f326409da20 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 --2- 172.21.15.138:0/2854622802 >> v2:172.21.15.190:6800/1562782189 conn(0x7f324403ce50 0x7f324403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(366..366 src has 1..366) ==== 5005+0+0 (secure 0 0 0) 0x7f325c04f3e0 con 0x7f326409da20 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 10 client.64470.objecter ms_dispatch 0x7f3264000ba0 osd_map(366..366 src has 1..366) 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 3 client.64470.objecter handle_osd_map got epochs [366,366] > 0 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 3 client.64470.objecter handle_osd_map decoding full epoch 366 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 20 client.64470.objecter dump_active .. 0 homeless 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3268a51640 1 --2- 172.21.15.138:0/2854622802 >> v2:172.21.15.190:6800/1562782189 conn(0x7f324403ce50 0x7f324403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3268a51640 10 monclient: get_auth_request con 0x7f324403ce50 auth_method 0 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 10 client.64470.objecter ms_handle_connect 0x7f3264070f90 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 10 client.64470.objecter resend_mon_ops 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264070f90 msgr2=0x7f3264071370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f32637fe640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264070f90 0x7f3264071370 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f324c0ae000 tx=0x7f324c0191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c0b1040 con 0x7f3264070f90 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 10 monclient: dump: 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f324c0b6640 con 0x7f3264070f90 2024-09-11T05:16:35.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:35.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:35.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264070f90 msgr2=0x7f3264071370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264070f90 0x7f3264071370 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f324c0ae000 tx=0x7f324c0191c0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264070f90 msgr2=0x7f3264071370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:35.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.725+0000 7f3260ff9640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264070f90 0x7f3264071370 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f324c0ae000 tx=0x7f324c0191c0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.729+0000 7f3260ff9640 10 client.64470.objecter ms_handle_connect 0x7f324403ce50 2024-09-11T05:16:35.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.729+0000 7f3268a51640 1 --2- 172.21.15.138:0/2854622802 >> v2:172.21.15.190:6800/1562782189 conn(0x7f324403ce50 0x7f324403f310 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7f3264072a60 tx=0x7f3254006c70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f326a4b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f326a4b4640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264133d70 0x7f3264134150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f326a4b4640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f32640a19a0 con 0x7f3264133d70 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264133d70 0x7f3264134150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 10 monclient: get_auth_request con 0x7f3264133d70 auth_method 0 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 10 monclient(hunting): handle_auth_done global_id 64476 payload 824 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.733+0000 7f32637fe640 1 -- 172.21.15.138:0/2854622802 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f326406cca0 con 0x7f3264133d70 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f32637fe640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264133d70 msgr2=0x7f3264134150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 10 client.64470.objecter ms_handle_connect 0x7f3264133d70 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 10 client.64470.objecter resend_mon_ops 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f32637fe640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264133d70 0x7f3264134150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f324c01c250 tx=0x7f324c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c0c3850 con 0x7f3264133d70 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 10 monclient: dump: 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.753 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f324c003620 con 0x7f3264133d70 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264133d70 msgr2=0x7f3264134150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264133d70 0x7f3264134150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f324c01c250 tx=0x7f324c009240 comp rx=0 tx=0).stop 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264133d70 msgr2=0x7f3264134150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:35.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3260ff9640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3264133d70 0x7f3264134150 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f324c01c250 tx=0x7f324c009240 comp rx=0 tx=0).stop 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 36, 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 299367, 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:16:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:16:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:16:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:16:35.768 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:16:35.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 10 librados: watch_flush enter 2024-09-11T05:16:35.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 10 librados: watch_flush exit 2024-09-11T05:16:35.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 20 client.64470.objecter shutdown clearing up homeless session... 2024-09-11T05:16:35.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 10 client.64470.objecter successfully canceled tick 2024-09-11T05:16:35.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 >> v2:172.21.15.190:6800/1562782189 conn(0x7f324403ce50 msgr2=0x7f324403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 --2- 172.21.15.138:0/2854622802 >> v2:172.21.15.190:6800/1562782189 conn(0x7f324403ce50 0x7f324403f310 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7f3264072a60 tx=0x7f3254006c70 comp rx=0 tx=0).stop 2024-09-11T05:16:35.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 10 monclient: shutdown 2024-09-11T05:16:35.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:35.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f326409da20 msgr2=0x7f326413dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 --2- 172.21.15.138:0/2854622802 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f326409da20 0x7f326413dcd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f325c00c820 tx=0x7f325c00cce0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3268a51640 1 -- 172.21.15.138:0/2854622802 reap_dead start 2024-09-11T05:16:35.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f3268a51640 1 -- 172.21.15.138:0/2854622802 reap_dead start 2024-09-11T05:16:35.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 shutdown_connections 2024-09-11T05:16:35.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 >> 172.21.15.138:0/2854622802 conn(0x7f3264063640 msgr2=0x7f3264133360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:35.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 shutdown_connections 2024-09-11T05:16:35.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 -- 172.21.15.138:0/2854622802 wait complete. 2024-09-11T05:16:35.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.737+0000 7f326a4b4640 1 librados: shutdown 2024-09-11T05:16:35.772 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:16:35.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T05:16:35.773 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:16:35.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.881+0000 7fa771ec2640 10 monclient: get_monmap_and_config 2024-09-11T05:16:35.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.881+0000 7fa771ec2640 10 monclient: build_initial_monmap 2024-09-11T05:16:35.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.881+0000 7fa771ec2640 10 monclient: monmap: 2024-09-11T05:16:35.890 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:35.890 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.890 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:16:35.889289+0000 2024-09-11T05:16:35.890 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:16:35.889289+0000 2024-09-11T05:16:35.891 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:35.891 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.891 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:16:35.891 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:16:35.891 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:16:35.891 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:16:35.892 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:16:35.892 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 1 Processor -- start 2024-09-11T05:16:35.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 1 -- start start 2024-09-11T05:16:35.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient: init 2024-09-11T05:16:35.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:35.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T05:16:35.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa76c09da20 0x7fa76c0a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient(hunting): picked mon.noname-a con 0x7fa76c09da20 addr v2:172.21.15.138:3300/0 2024-09-11T05:16:35.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa76c0a1e60 addr v1:172.21.15.138:6789/0 2024-09-11T05:16:35.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa76c0a22d0 0x7fa76c136910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient(hunting): picked mon.noname-c con 0x7fa76c0a22d0 addr v2:172.21.15.190:3300/0 2024-09-11T05:16:35.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa76c064590 con 0x7fa76c0a1e60 2024-09-11T05:16:35.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fa76c09de00 con 0x7fa76c09da20 2024-09-11T05:16:35.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa76c0a2c70 con 0x7fa76c0a22d0 2024-09-11T05:16:35.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:35.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa771ec2640 10 monclient(hunting): authenticate will time out at 1479.325055s 2024-09-11T05:16:35.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa76c09da20 0x7fa76c0a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76affd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fa76c0a1e60 0x7fa76c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:40614/0 (socket says 172.21.15.138:40614) 2024-09-11T05:16:35.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76affd640 1 -- 172.21.15.138:0/1225058648 learned_addr learned my addr 172.21.15.138:0/1225058648 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:16:35.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76bfff640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7fa76c0a22d0 msgr2=0x7fa76c136910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:16:35.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient(hunting): get_auth_request con 0x7fa76c09da20 auth_method 0 2024-09-11T05:16:35.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76bfff640 1 --2- 172.21.15.138:0/1225058648 >> v2:172.21.15.190:3300/0 conn(0x7fa76c0a22d0 0x7fa76c136910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:35.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_done global_id 64479 payload 274 2024-09-11T05:16:35.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 1 -- 172.21.15.138:0/1225058648 >> v2:172.21.15.190:3300/0 conn(0x7fa76c0a22d0 msgr2=0x7fa76c136910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:16:35.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 1 --2- 172.21.15.138:0/1225058648 >> v2:172.21.15.190:3300/0 conn(0x7fa76c0a22d0 0x7fa76c136910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 1 -- 172.21.15.138:0/1225058648 >> v1:172.21.15.138:6789/0 conn(0x7fa76c0a1e60 legacy=0x7fa76c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:16:35.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:16:35.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 1 monclient: found mon.noname-a 2024-09-11T05:16:35.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:35.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:35.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.885+0000 7fa76b7fe640 1 -- 172.21.15.138:0/1225058648 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c0a2f50 con 0x7fa76c09da20 2024-09-11T05:16:35.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76b7fe640 1 -- 172.21.15.138:0/1225058648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c09da20 msgr2=0x7fa76c0a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76b7fe640 1 --2- 172.21.15.138:0/1225058648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c09da20 0x7fa76c0a1920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0a2b60 tx=0x7fa75c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b5f6ad57d62bc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 1 -- 172.21.15.138:0/1225058648 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c0b0070 con 0x7fa76c09da20 2024-09-11T05:16:35.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:35.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: dump: 2024-09-11T05:16:35.919 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.919 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.919 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.920 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.920 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.920 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.920 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.920 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.921 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.921 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: _finish_auth 0 2024-09-11T05:16:35.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:16:35.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:35.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 1 -- 172.21.15.138:0/1225058648 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa75c0ab600 con 0x7fa76c09da20 2024-09-11T05:16:35.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:35.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 1 -- 172.21.15.138:0/1225058648 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c0ab9d0 con 0x7fa76c09da20 2024-09-11T05:16:35.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:35.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa76a7fc640 10 monclient: dump: 2024-09-11T05:16:35.922 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.923 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.923 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.923 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.923 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.923 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.924 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.924 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.924 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.924 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 5 monclient: authenticate success, global_id 64479 2024-09-11T05:16:35.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 10 monclient: get_monmap_and_config success 2024-09-11T05:16:35.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 4 set_mon_vals no callback set 2024-09-11T05:16:35.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 10 monclient: shutdown 2024-09-11T05:16:35.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:35.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 1 -- 172.21.15.138:0/1225058648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c09da20 msgr2=0x7fa76c0a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 1 --2- 172.21.15.138:0/1225058648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c09da20 0x7fa76c0a1920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0a2b60 tx=0x7fa75c01c320 comp rx=0 tx=0).stop 2024-09-11T05:16:35.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 1 -- 172.21.15.138:0/1225058648 shutdown_connections 2024-09-11T05:16:35.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 1 --2- 172.21.15.138:0/1225058648 >> v2:172.21.15.190:3300/0 conn(0x7fa76c0a22d0 0x7fa76c136910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 1 --2- 172.21.15.138:0/1225058648 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c09da20 0x7fa76c0a1920 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 1 -- 172.21.15.138:0/1225058648 >> 172.21.15.138:0/1225058648 conn(0x7fa76c063640 msgr2=0x7fa76c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:35.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 1 -- 172.21.15.138:0/1225058648 shutdown_connections 2024-09-11T05:16:35.926 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.889+0000 7fa771ec2640 1 -- 172.21.15.138:0/1225058648 wait complete. 2024-09-11T05:16:35.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient: build_initial_monmap 2024-09-11T05:16:35.927 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient: monmap: 2024-09-11T05:16:35.927 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:16:35.927 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:16:35.927 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:16:35.927 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:16:35.927 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 librados: starting msgr at 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 librados: starting objecter 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 Processor -- start 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 -- start start 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 librados: setting wanted keys 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 librados: calling monclient init 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient: init 2024-09-11T05:16:35.928 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient: _reopen_session rank -1 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa76c09da20 0x7fa76c1d0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient(hunting): picked mon.noname-b con 0x7fa76c09da20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c0a1e60 0x7fa76c13da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient(hunting): picked mon.noname-c con 0x7fa76c0a1e60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c0a22d0 0x7fa76c13df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient(hunting): picked mon.noname-a con 0x7fa76c0a22d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c140ee0 con 0x7fa76c0a22d0 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c140d60 con 0x7fa76c09da20 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa76c141060 con 0x7fa76c0a1e60 2024-09-11T05:16:35.929 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient(hunting): _renew_subs 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa771ec2640 10 monclient(hunting): authenticate will time out at 1479.333186s 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c0a22d0 0x7fa76c13df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c0a1e60 0x7fa76c13da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76b7fe640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa76c09da20 msgr2=0x7fa76c1d0550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76b7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa76c09da20 0x7fa76c1d0550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c0a22d0 0x7fa76c13df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:48840/0 (socket says 172.21.15.138:48840) 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c0a1e60 0x7fa76c13da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59660/0 (socket says 172.21.15.138:59660) 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 -- 172.21.15.138:0/4072101158 learned_addr learned my addr 172.21.15.138:0/4072101158 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient(hunting): get_auth_request con 0x7fa76c0a22d0 auth_method 0 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 10 monclient(hunting): get_auth_request con 0x7fa76c0a1e60 auth_method 0 2024-09-11T05:16:35.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient(hunting): handle_auth_done global_id 64482 payload 824 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c0a1e60 msgr2=0x7fa76c13da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c0a1e60 0x7fa76c13da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa76c09da20 msgr2=0x7fa76c1d0550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:16:35.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa76c09da20 0x7fa76c1d0550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 monclient: found mon.noname-a 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 -- 172.21.15.138:0/4072101158 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c1d0a90 con 0x7fa76c0a22d0 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76affd640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c0a1e60 0x7fa76c13da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c0a22d0 msgr2=0x7fa76c13df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa76bfff640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c0a22d0 0x7fa76c13df40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa76400be80 tx=0x7fa76400beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa753fff640 10 client.?.objecter ms_handle_connect 0x7fa76c0a22d0 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.893+0000 7fa753fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764016630 con 0x7fa76c0a22d0 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:16:35.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: dump: 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: _finish_auth 0 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: _check_auth_tickets 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:16:35.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa764010030 con 0x7fa76c0a22d0 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764016a10 con 0x7fa76c0a22d0 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 monclient: dump: 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.934 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa769ffb640 4 set_mon_vals no callback set 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 5 monclient: authenticate success, global_id 64482 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 10 monclient: _renew_subs 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76c0a0190 con 0x7fa76c0a22d0 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 10 client.64482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 10 monclient: _renew_subs 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76c137ca0 con 0x7fa76c0a22d0 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 1 librados: init done 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c1321c0 0x7fa76c1325a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa76c09e420 con 0x7fa76c1321c0 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c1321c0 0x7fa76c1325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 10 monclient: get_auth_request con 0x7fa76c1321c0 auth_method 0 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_done global_id 84155 payload 824 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fa7640153e0 con 0x7fa76c0a22d0 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 1 -- 172.21.15.138:0/4072101158 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa76c139050 con 0x7fa76c1321c0 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 1 --2- 172.21.15.138:0/4072101158 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa75803ce50 0x7fa75803f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(366..366 src has 1..366) ==== 5005+0+0 (secure 0 0 0) 0x7fa764050290 con 0x7fa76c0a22d0 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 client.64482.objecter ms_dispatch 0x7fa76c000ba0 osd_map(366..366 src has 1..366) 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 3 client.64482.objecter handle_osd_map got epochs [366,366] > 0 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 3 client.64482.objecter handle_osd_map decoding full epoch 366 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 20 client.64482.objecter dump_active .. 0 homeless 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76affd640 1 --2- 172.21.15.138:0/4072101158 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa75803ce50 0x7fa75803f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76affd640 10 monclient: get_auth_request con 0x7fa75803ce50 auth_method 0 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 client.64482.objecter ms_handle_connect 0x7fa76c1321c0 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa753fff640 10 client.64482.objecter resend_mon_ops 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c1321c0 msgr2=0x7fa76c1325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.897+0000 7fa76b7fe640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c1321c0 0x7fa76c1325a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa76c09e050 tx=0x7fa75c01ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c0bb020 con 0x7fa76c1321c0 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:35.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 10 monclient: dump: 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa75c0b1040 con 0x7fa76c1321c0 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:16:35.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c1321c0 msgr2=0x7fa76c1325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c1321c0 0x7fa76c1325a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa76c09e050 tx=0x7fa75c01ab60 comp rx=0 tx=0).stop 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c1321c0 msgr2=0x7fa76c1325a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c1321c0 0x7fa76c1325a0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa76c09e050 tx=0x7fa75c01ab60 comp rx=0 tx=0).stop 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa76affd640 1 --2- 172.21.15.138:0/4072101158 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa75803ce50 0x7fa75803f310 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7fa760007ca0 tx=0x7fa760007bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.901+0000 7fa753fff640 10 client.64482.objecter ms_handle_connect 0x7fa75803ce50 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa771ec2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa771ec2640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c132ae0 0x7fa76c134a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa771ec2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa76c0a19a0 con 0x7fa76c132ae0 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c132ae0 0x7fa76c134a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 10 monclient: get_auth_request con 0x7fa76c132ae0 auth_method 0 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:16:35.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 10 monclient(hunting): handle_auth_done global_id 84158 payload 824 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.905+0000 7fa76b7fe640 1 -- 172.21.15.138:0/4072101158 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa76c09e110 con 0x7fa76c132ae0 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 10 client.64482.objecter ms_handle_connect 0x7fa76c132ae0 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 10 client.64482.objecter resend_mon_ops 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa76b7fe640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c132ae0 msgr2=0x7fa76c134a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa76b7fe640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c132ae0 0x7fa76c134a70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa75c01a9d0 tx=0x7fa75c0190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c0bb040 con 0x7fa76c132ae0 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:16:35.940 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 10 monclient: dump: 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7fa75c000e50 con 0x7fa76c132ae0 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:16:35.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c132ae0 msgr2=0x7fa76c134a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c132ae0 0x7fa76c134a70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa75c01a9d0 tx=0x7fa75c0190c0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c132ae0 msgr2=0x7fa76c134a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:16:35.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa753fff640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa76c132ae0 0x7fa76c134a70 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa75c01a9d0 tx=0x7fa75c0190c0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.942 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:16:35.942 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:35.942 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:35.942 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:16:35.942 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 36, 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 39679, 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:35.943 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:35.944 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:16:35.945 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:16:35.946 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:16:35.947 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.948 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:35.949 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:35.950 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:16:35.951 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:16:35.952 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:16:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:16:35.954 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:16:35.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 10 librados: watch_flush enter 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 10 librados: watch_flush exit 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 20 client.64482.objecter shutdown clearing up homeless session... 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 10 client.64482.objecter successfully canceled tick 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa75803ce50 msgr2=0x7fa75803f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 --2- 172.21.15.138:0/4072101158 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa75803ce50 0x7fa75803f310 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7fa760007ca0 tx=0x7fa760007bf0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 10 monclient: shutdown 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c0a22d0 msgr2=0x7fa76c13df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 --2- 172.21.15.138:0/4072101158 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa76c0a22d0 0x7fa76c13df40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa76400be80 tx=0x7fa76400beb0 comp rx=0 tx=0).stop 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa76bfff640 1 -- 172.21.15.138:0/4072101158 reap_dead start 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa76bfff640 1 -- 172.21.15.138:0/4072101158 reap_dead start 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 shutdown_connections 2024-09-11T05:16:35.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 >> 172.21.15.138:0/4072101158 conn(0x7fa76c063640 msgr2=0x7fa76c12fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:16:35.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 shutdown_connections 2024-09-11T05:16:35.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 -- 172.21.15.138:0/4072101158 wait complete. 2024-09-11T05:16:35.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:16:35.909+0000 7fa771ec2640 1 librados: shutdown 2024-09-11T05:16:35.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:16:35.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:17:08.279 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T05:17:08.279+0000 7faf59caf640 -1 snap_mapper.add_oid found existing snaps mapped on 300:afb7b488:test-rados-api-smithi190-28187-80::foo:2, removing 2024-09-11T05:17:08.280 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T05:17:08.283+0000 7ff604faa640 -1 snap_mapper.add_oid found existing snaps mapped on 300:afb7b488:test-rados-api-smithi190-28187-80::foo:2, removing 2024-09-11T05:17:39.290 INFO:tasks.workunit.client.0.smithi190.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18050 ms) 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22969 ms) 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3056 ms) 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (2992 ms) 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (17957 ms) 2024-09-11T05:17:39.291 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3480 ms) 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4040 ms) 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38497 ms) 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17172 ms) 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.292 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3855 ms) 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3030 ms) 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24401 ms) 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (34356 ms) 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5016 ms) 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-11T05:17:39.293 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3909 ms) 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: waiting for scrubs... 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: done waiting 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25394 ms) 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9728 ms) 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9058 ms) 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-11T05:17:39.294 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:17:39.295 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8881 ms) 2024-09-11T05:17:39.295 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-11T05:18:05.958 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T05:18:05.958 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T05:18:05.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T05:18:05.962 INFO:tasks.ceph.mon.b:Started 2024-09-11T05:18:05.962 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-11T05:18:05.962 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-11T05:18:05.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T05:18:20.963 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-11T05:18:20.964 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-11T05:18:20.964 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:18:20.964 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:18:21.098 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient: get_monmap_and_config 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient: build_initial_monmap 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient: monmap: 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:21.103409+0000 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:21.103409+0000 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:21.099 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:21.100 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:21.100 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 1 Processor -- start 2024-09-11T05:18:21.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 1 -- start start 2024-09-11T05:18:21.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient: init 2024-09-11T05:18:21.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-11T05:18:21.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b0806e0e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:21.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3b0809eb30 0x7f3b0813a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient(hunting): picked mon.noname-a con 0x7f3b0809eb30 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809ef40 0x7f3b0813ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient(hunting): picked mon.noname-b con 0x7f3b0809ef40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0809f540 con 0x7f3b0806e0e0 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f3b0806e4c0 con 0x7f3b0809eb30 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0809f820 con 0x7f3b0809ef40 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0ea30640 10 monclient(hunting): authenticate will time out at 1584.539402s 2024-09-11T05:18:21.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3b0809eb30 0x7f3b0813a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809ef40 0x7f3b0813ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b07fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f3b0806e0e0 0x7f3b0809e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38398/0 (socket says 172.21.15.138:38398) 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 10 monclient(hunting): get_auth_request con 0x7f3b0809eb30 auth_method 0 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b07fff640 1 -- 172.21.15.138:0/1333605250 learned_addr learned my addr 172.21.15.138:0/1333605250 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient(hunting): get_auth_request con 0x7f3b0809ef40 auth_method 0 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient(hunting): handle_auth_done global_id 84161 payload 274 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 1 -- 172.21.15.138:0/1333605250 >> v2:172.21.15.138:3300/0 conn(0x7f3b0809eb30 msgr2=0x7f3b0813a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 1 --2- 172.21.15.138:0/1333605250 >> v2:172.21.15.138:3300/0 conn(0x7f3b0809eb30 0x7f3b0813a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 1 -- 172.21.15.138:0/1333605250 >> v1:172.21.15.138:6789/0 conn(0x7f3b0806e0e0 legacy=0x7f3b0809e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 1 monclient: found mon.noname-b 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b0cfcd640 1 -- 172.21.15.138:0/1333605250 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0809fb00 con 0x7f3b0809ef40 2024-09-11T05:18:21.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.100+0000 7f3b077fe640 1 --2- 172.21.15.138:0/1333605250 >> v2:172.21.15.138:3300/0 conn(0x7f3b0809eb30 0x7f3b0813a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:21.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0cfcd640 1 -- 172.21.15.138:0/1333605250 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809ef40 msgr2=0x7f3b0813ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0cfcd640 1 --2- 172.21.15.138:0/1333605250 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809ef40 0x7f3b0813ce80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3af80097f0 tx=0x7f3af801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea3a12d36c6f6e08 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 1 -- 172.21.15.138:0/1333605250 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80b0070 con 0x7f3b0809ef40 2024-09-11T05:18:21.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:18:21.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: dump: 2024-09-11T05:18:21.112 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.112 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 1 -- 172.21.15.138:0/1333605250 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af80ab600 con 0x7f3b0809ef40 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 5 monclient: authenticate success, global_id 84161 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 1 -- 172.21.15.138:0/1333605250 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80ab9d0 con 0x7f3b0809ef40 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 4 set_mon_vals no callback set 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b06ffd640 10 monclient: dump: 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.114 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient: shutdown 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- 172.21.15.138:0/1333605250 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809ef40 msgr2=0x7f3b0813ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/1333605250 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809ef40 0x7f3b0813ce80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3af80097f0 tx=0x7f3af801c360 comp rx=0 tx=0).stop 2024-09-11T05:18:21.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- 172.21.15.138:0/1333605250 shutdown_connections 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/1333605250 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809ef40 0x7f3b0813ce80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/1333605250 >> v2:172.21.15.138:3300/0 conn(0x7f3b0809eb30 0x7f3b0813a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- 172.21.15.138:0/1333605250 >> 172.21.15.138:0/1333605250 conn(0x7f3b08063640 msgr2=0x7f3b08063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- 172.21.15.138:0/1333605250 shutdown_connections 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- 172.21.15.138:0/1333605250 wait complete. 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient: build_initial_monmap 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient: monmap: 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:21.116 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 librados: starting msgr at 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 librados: starting objecter 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 Processor -- start 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- start start 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 librados: setting wanted keys 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 librados: calling monclient init 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient: init 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b0806e0e0 0x7f3b0813b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient(hunting): picked mon.noname-a con 0x7f3b0806e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809eb30 0x7f3b0813ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient(hunting): picked mon.noname-c con 0x7f3b0809eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b0809ef40 0x7f3b0813bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient(hunting): picked mon.noname-b con 0x7f3b0809ef40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3b080a2330 con 0x7f3b0806e0e0 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3b080a21b0 con 0x7f3b0809ef40 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3b080a24b0 con 0x7f3b0809eb30 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0ea30640 10 monclient(hunting): authenticate will time out at 1584.546362s 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809eb30 0x7f3b0813ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b0806e0e0 0x7f3b0813b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809eb30 0x7f3b0813ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36198/0 (socket says 172.21.15.138:36198) 2024-09-11T05:18:21.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 -- 172.21.15.138:0/158067481 learned_addr learned my addr 172.21.15.138:0/158067481 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient(hunting): get_auth_request con 0x7f3b0809eb30 auth_method 0 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 10 monclient(hunting): get_auth_request con 0x7f3b0806e0e0 auth_method 0 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0cfcd640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b0809ef40 0x7f3b0813bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0cfcd640 10 monclient(hunting): get_auth_request con 0x7f3b0809ef40 auth_method 0 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0cfcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0cfcd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b0cfcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient(hunting): handle_auth_done global_id 84164 payload 824 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 -- 172.21.15.138:0/158067481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b0809ef40 msgr2=0x7f3b0813bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b0809ef40 0x7f3b0813bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 -- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b0806e0e0 msgr2=0x7f3b0813b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b0806e0e0 0x7f3b0813b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 monclient: found mon.noname-c 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b077fe640 1 -- 172.21.15.138:0/158067481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0811ed50 con 0x7f3b0809eb30 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.108+0000 7f3b07fff640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b0806e0e0 0x7f3b0813b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b077fe640 1 -- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809eb30 msgr2=0x7f3b0813ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b077fe640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809eb30 0x7f3b0813ba50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3af000efc0 tx=0x7f3af000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 client.?.objecter ms_handle_connect 0x7f3b0809eb30 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 1 -- 172.21.15.138:0/158067481 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af0018020 con 0x7f3b0809eb30 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: dump: 2024-09-11T05:18:21.121 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 1 -- 172.21.15.138:0/158067481 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af0014890 con 0x7f3b0809eb30 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 5 monclient: authenticate success, global_id 84164 2024-09-11T05:18:21.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 10 monclient: _renew_subs 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0811ffe0 con 0x7f3b0809eb30 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 1 -- 172.21.15.138:0/158067481 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af0014c60 con 0x7f3b0809eb30 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 monclient: dump: 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.123 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 10 client.84164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 10 monclient: _renew_subs 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b0811fbc0 con 0x7f3b0809eb30 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b067fc640 4 set_mon_vals no callback set 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 1 librados: init done 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b0809e420 con 0x7f3b0809eb30 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 1 -- 172.21.15.138:0/158067481 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3af001d400 con 0x7f3b0809eb30 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 1 --2- 172.21.15.138:0/158067481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3ae803ced0 0x7f3ae803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 1 -- 172.21.15.138:0/158067481 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f3af0050cc0 con 0x7f3b0809eb30 2024-09-11T05:18:21.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 10 client.84164.objecter ms_dispatch 0x7f3b08000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:21.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 3 client.84164.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:21.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 3 client.84164.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:21.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b04ff9640 20 client.84164.objecter dump_active .. 0 homeless 2024-09-11T05:18:21.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b07fff640 1 --2- 172.21.15.138:0/158067481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3ae803ced0 0x7f3ae803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.112+0000 7f3b07fff640 10 monclient: get_auth_request con 0x7f3ae803ced0 auth_method 0 2024-09-11T05:18:21.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.116+0000 7f3b04ff9640 10 client.84164.objecter ms_handle_connect 0x7f3ae803ced0 2024-09-11T05:18:21.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.116+0000 7f3b07fff640 1 --2- 172.21.15.138:0/158067481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3ae803ced0 0x7f3ae803f390 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009e00 tx=0x7f3b000098c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.116+0000 7f3b04ff9640 1 -- 172.21.15.138:0/158067481 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3af0022030 con 0x7f3b0809eb30 2024-09-11T05:18:21.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.116+0000 7f3b04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.116+0000 7f3b04ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:21.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.352+0000 7f3b0ea30640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:18:21.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.352+0000 7f3b0ea30640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:21.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.352+0000 7f3b0ea30640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.352+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3b0809dea0 con 0x7f3b0809eb30 2024-09-11T05:18:21.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.352+0000 7f3b04ff9640 1 -- 172.21.15.138:0/158067481 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f3b0811ffe0 con 0x7f3b0809eb30 2024-09-11T05:18:21.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.352+0000 7f3b04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:21.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.352+0000 7f3b04ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:21.352 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:18:21.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 10 librados: watch_flush enter 2024-09-11T05:18:21.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 10 librados: watch_flush exit 2024-09-11T05:18:21.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 20 client.84164.objecter shutdown clearing up homeless session... 2024-09-11T05:18:21.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 10 client.84164.objecter successfully canceled tick 2024-09-11T05:18:21.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3ae803ced0 msgr2=0x7f3ae803f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/158067481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3ae803ced0 0x7f3ae803f390 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009e00 tx=0x7f3b000098c0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 10 monclient: shutdown 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809eb30 msgr2=0x7f3b0813ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809eb30 0x7f3b0813ba50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3af000efc0 tx=0x7f3af000c490 comp rx=0 tx=0).stop 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 shutdown_connections 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/158067481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3ae803ced0 0x7f3ae803f390 unknown :-1 s=CLOSED pgs=3432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b0809ef40 0x7f3b0813bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3b0809eb30 0x7f3b0813ba50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 --2- 172.21.15.138:0/158067481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3b0806e0e0 0x7f3b0813b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 >> 172.21.15.138:0/158067481 conn(0x7f3b08063640 msgr2=0x7f3b081327c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 shutdown_connections 2024-09-11T05:18:21.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 -- 172.21.15.138:0/158067481 wait complete. 2024-09-11T05:18:21.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.356+0000 7f3b0ea30640 1 librados: shutdown 2024-09-11T05:18:21.369 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:18:21.369 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:18:21.369 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:18:21.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient: get_monmap_and_config 2024-09-11T05:18:21.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient: build_initial_monmap 2024-09-11T05:18:21.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient: monmap: 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:21.493524+0000 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:21.493524+0000 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:21.489 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 1 Processor -- start 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 1 -- start start 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient: init 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3f88070010 0x7f3f88135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f88070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f8809e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f881382c0 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f880a8990 con 0x7f3f881382c0 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f3f880703f0 con 0x7f3f88070010 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8809eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36202/0 (socket says 172.21.15.138:36202) 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 1 -- 172.21.15.138:0/3534948010 learned_addr learned my addr 172.21.15.138:0/3534948010 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 1 --2- 172.21.15.138:0/3534948010 >> v2:172.21.15.138:3300/0 conn(0x7f3f88070010 0x7f3f88135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 1 -- 172.21.15.138:0/3534948010 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8809cbf0 con 0x7f3f8809e6e0 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8f68d640 10 monclient(hunting): authenticate will time out at 1584.930065s 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 10 monclient(hunting): get_auth_request con 0x7f3f8809e6e0 auth_method 0 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient(hunting): get_auth_request con 0x7f3f88070010 auth_method 0 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient(hunting): handle_auth_done global_id 64491 payload 274 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 1 -- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 msgr2=0x7f3f8809eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 1 --2- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8809eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 1 -- 172.21.15.138:0/3534948010 >> v1:172.21.15.138:6789/0 conn(0x7f3f881382c0 legacy=0x7f3f88138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 1 monclient: found mon.noname-a 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8d429640 1 -- 172.21.15.138:0/3534948010 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8809ced0 con 0x7f3f88070010 2024-09-11T05:18:21.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.492+0000 7f3f8cc28640 1 --2- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:21.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f8d429640 1 -- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 msgr2=0x7f3f88135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f8d429640 1 --2- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 0x7f3f88135450 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3f840097c0 tx=0x7f3f8401c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1b4931005b0083 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 1 -- 172.21.15.138:0/3534948010 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840b0070 con 0x7f3f88070010 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: dump: 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.501 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 1 -- 172.21.15.138:0/3534948010 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f840ab600 con 0x7f3f88070010 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 1 -- 172.21.15.138:0/3534948010 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840ab9d0 con 0x7f3f88070010 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f8f68d640 5 monclient: authenticate success, global_id 64491 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f8f68d640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f8f68d640 4 set_mon_vals no callback set 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:21.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.496+0000 7f3f7ffff640 10 monclient: dump: 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient: shutdown 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 msgr2=0x7f3f88135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 --2- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 0x7f3f88135450 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3f840097c0 tx=0x7f3f8401c330 comp rx=0 tx=0).stop 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- 172.21.15.138:0/3534948010 shutdown_connections 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 --2- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 --2- 172.21.15.138:0/3534948010 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 0x7f3f88135450 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- 172.21.15.138:0/3534948010 >> 172.21.15.138:0/3534948010 conn(0x7f3f880ab830 msgr2=0x7f3f880a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- 172.21.15.138:0/3534948010 shutdown_connections 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- 172.21.15.138:0/3534948010 wait complete. 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient: build_initial_monmap 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient: monmap: 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:21.504 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 librados: starting msgr at 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 librados: starting objecter 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 Processor -- start 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- start start 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 librados: setting wanted keys 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 librados: calling monclient init 2024-09-11T05:18:21.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient: init 2024-09-11T05:18:21.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 0x7f3f881d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f88070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8813f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f8809e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f881382c0 0x7f3f8813fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f881382c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881432f0 con 0x7f3f88070010 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3f88143170 con 0x7f3f881382c0 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3f88143470 con 0x7f3f8809e6e0 2024-09-11T05:18:21.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8f68d640 10 monclient(hunting): authenticate will time out at 1584.937868s 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 0x7f3f881d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 0x7f3f881d07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43434/0 (socket says 172.21.15.138:43434) 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8813f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36208/0 (socket says 172.21.15.138:36208) 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 -- 172.21.15.138:0/1657697553 learned_addr learned my addr 172.21.15.138:0/1657697553 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient(hunting): get_auth_request con 0x7f3f88070010 auth_method 0 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 10 monclient(hunting): get_auth_request con 0x7f3f8809e6e0 auth_method 0 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8dc2a640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f881382c0 0x7f3f8813fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8dc2a640 10 monclient(hunting): get_auth_request con 0x7f3f881382c0 auth_method 0 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8dc2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8dc2a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8dc2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8dc2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8dc2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8dc2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient(hunting): handle_auth_done global_id 64494 payload 824 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 msgr2=0x7f3f8813f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3f8809e6e0 0x7f3f8813f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f881382c0 msgr2=0x7f3f8813fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f881382c0 0x7f3f8813fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 monclient: found mon.noname-a 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.500+0000 7f3f8d429640 1 -- 172.21.15.138:0/1657697553 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8813b040 con 0x7f3f88070010 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8d429640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 msgr2=0x7f3f881d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8d429640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 0x7f3f881d07d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3f8401cbd0 tx=0x7f3f840190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 client.?.objecter ms_handle_connect 0x7f3f88070010 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840b8020 con 0x7f3f88070010 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: dump: 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.515 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f840b4620 con 0x7f3f88070010 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840b0040 con 0x7f3f88070010 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 monclient: dump: 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.516 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7effd640 4 set_mon_vals no callback set 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 5 monclient: authenticate success, global_id 64494 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 10 monclient: _renew_subs 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f88140300 con 0x7f3f88070010 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 10 client.64494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 10 monclient: _renew_subs 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:21.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f8813ad60 con 0x7f3f88070010 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 1 librados: init done 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a3930 0x7f3f880a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8806fdb0 con 0x7f3f880a3930 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a3930 0x7f3f880a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 10 monclient: get_auth_request con 0x7f3f880a3930 auth_method 0 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3f840b3050 con 0x7f3f88070010 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_done global_id 64497 payload 824 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8cc28640 1 -- 172.21.15.138:0/1657697553 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f88135450 con 0x7f3f880a3930 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 1 --2- 172.21.15.138:0/1657697553 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3f6c03cea0 0x7f3f6c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f3f840e96c0 con 0x7f3f88070010 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 10 client.64494.objecter ms_dispatch 0x7f3f88000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 3 client.64494.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 3 client.64494.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f7dffb640 20 client.64494.objecter dump_active .. 0 homeless 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8dc2a640 1 --2- 172.21.15.138:0/1657697553 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3f6c03cea0 0x7f3f6c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.504+0000 7f3f8dc2a640 10 monclient: get_auth_request con 0x7f3f6c03cea0 auth_method 0 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 10 client.64494.objecter ms_handle_connect 0x7f3f880a3930 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f8cc28640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a3930 msgr2=0x7f3f880a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 10 client.64494.objecter resend_mon_ops 2024-09-11T05:18:21.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f8cc28640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a3930 0x7f3f880a3d10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3f78006cd0 tx=0x7f3f78011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800fd60 con 0x7f3f880a3930 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 10 monclient: dump: 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.520 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3f7800fd60 con 0x7f3f880a3930 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a3930 msgr2=0x7f3f880a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a3930 0x7f3f880a3d10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3f78006cd0 tx=0x7f3f78011d80 comp rx=0 tx=0).stop 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a3930 msgr2=0x7f3f880a3d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a3930 0x7f3f880a3d10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f7dffb640 10 client.64494.objecter ms_handle_connect 0x7f3f6c03cea0 2024-09-11T05:18:21.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.508+0000 7f3f8dc2a640 1 --2- 172.21.15.138:0/1657697553 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3f6c03cea0 0x7f3f6c03f360 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f3f88139410 tx=0x7f3f80003d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8f68d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:18:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8f68d640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a4250 0x7f3f8809f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8f68d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3f880a2330 con 0x7f3f880a4250 2024-09-11T05:18:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a4250 0x7f3f8809f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 10 monclient: get_auth_request con 0x7f3f880a4250 auth_method 0 2024-09-11T05:18:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 10 monclient(hunting): handle_auth_done global_id 64500 payload 824 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.512+0000 7f3f8cc28640 1 -- 172.21.15.138:0/1657697553 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f88069fa0 con 0x7f3f880a4250 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8cc28640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a4250 msgr2=0x7f3f8809f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8cc28640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a4250 0x7f3f8809f030 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3f78006cd0 tx=0x7f3f780046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 10 client.64494.objecter ms_handle_connect 0x7f3f880a4250 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 10 client.64494.objecter resend_mon_ops 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78014a60 con 0x7f3f880a4250 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 10 monclient: dump: 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f3f78000e50 con 0x7f3f880a4250 2024-09-11T05:18:21.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a4250 msgr2=0x7f3f8809f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a4250 0x7f3f8809f030 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3f78006cd0 tx=0x7f3f780046d0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a4250 msgr2=0x7f3f8809f030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f7dffb640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f880a4250 0x7f3f8809f030 secure :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3f78006cd0 tx=0x7f3f780046d0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 40, 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 405146, 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:18:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:18:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:18:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:18:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:18:21.539 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 10 librados: watch_flush enter 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 10 librados: watch_flush exit 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 20 client.64494.objecter shutdown clearing up homeless session... 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 10 client.64494.objecter successfully canceled tick 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3f6c03cea0 msgr2=0x7f3f6c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 --2- 172.21.15.138:0/1657697553 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3f6c03cea0 0x7f3f6c03f360 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f3f88139410 tx=0x7f3f80003d50 comp rx=0 tx=0).stop 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 10 monclient: shutdown 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 msgr2=0x7f3f881d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 --2- 172.21.15.138:0/1657697553 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3f88070010 0x7f3f881d07d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3f8401cbd0 tx=0x7f3f840190d0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8dc2a640 1 -- 172.21.15.138:0/1657697553 reap_dead start 2024-09-11T05:18:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8dc2a640 1 -- 172.21.15.138:0/1657697553 reap_dead start 2024-09-11T05:18:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 shutdown_connections 2024-09-11T05:18:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 >> 172.21.15.138:0/1657697553 conn(0x7f3f880ab830 msgr2=0x7f3f880a97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 shutdown_connections 2024-09-11T05:18:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 -- 172.21.15.138:0/1657697553 wait complete. 2024-09-11T05:18:21.542 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.516+0000 7f3f8f68d640 1 librados: shutdown 2024-09-11T05:18:21.542 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:18:21.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:18:21.542 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:18:21.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient: get_monmap_and_config 2024-09-11T05:18:21.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient: build_initial_monmap 2024-09-11T05:18:21.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient: monmap: 2024-09-11T05:18:21.664 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:21.669452+0000 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:21.669452+0000 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:21.665 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 1 Processor -- start 2024-09-11T05:18:21.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 1 -- start start 2024-09-11T05:18:21.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient: init 2024-09-11T05:18:21.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c09e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient(hunting): picked mon.noname-b con 0x7f7e3c0a32c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f7e3c09e960 0x7f7e3c09ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3c09e960 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e3c1356f0 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e3c0a1b50 con 0x7f7e3c1356f0 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f7e3c09fe70 con 0x7f7e3c09e960 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c0a1920 con 0x7f7e3c0a32c0 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e414d6640 10 monclient(hunting): authenticate will time out at 1585.105605s 2024-09-11T05:18:21.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c09e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c09e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36220/0 (socket says 172.21.15.138:36220) 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 1 -- 172.21.15.138:0/1230356440 learned_addr learned my addr 172.21.15.138:0/1230356440 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient(hunting): get_auth_request con 0x7f7e3c0a32c0 auth_method 0 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3a7fc640 1 --2- 172.21.15.138:0/1230356440 >> v2:172.21.15.138:3300/0 conn(0x7f7e3c09e960 0x7f7e3c09ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3a7fc640 10 monclient(hunting): get_auth_request con 0x7f7e3c09e960 auth_method 0 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient(hunting): handle_auth_done global_id 84173 payload 274 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 1 -- 172.21.15.138:0/1230356440 >> v2:172.21.15.138:3300/0 conn(0x7f7e3c09e960 msgr2=0x7f7e3c09ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 1 --2- 172.21.15.138:0/1230356440 >> v2:172.21.15.138:3300/0 conn(0x7f7e3c09e960 0x7f7e3c09ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 1 -- 172.21.15.138:0/1230356440 >> v1:172.21.15.138:6789/0 conn(0x7f7e3c1356f0 legacy=0x7f7e3c135ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 1 monclient: found mon.noname-b 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.668+0000 7f7e3affd640 1 -- 172.21.15.138:0/1230356440 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c09f560 con 0x7f7e3c0a32c0 2024-09-11T05:18:21.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e3affd640 1 -- 172.21.15.138:0/1230356440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 msgr2=0x7f7e3c09e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e3affd640 1 --2- 172.21.15.138:0/1230356440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c09e420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009870 tx=0x7f7e2c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b507484e8e1dd912 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 1 -- 172.21.15.138:0/1230356440 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c0b0070 con 0x7f7e3c0a32c0 2024-09-11T05:18:21.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:18:21.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: dump: 2024-09-11T05:18:21.675 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.675 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 1 -- 172.21.15.138:0/1230356440 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e2c0ab600 con 0x7f7e3c0a32c0 2024-09-11T05:18:21.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 5 monclient: authenticate success, global_id 84173 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 1 -- 172.21.15.138:0/1230356440 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c0a4e60 con 0x7f7e3c0a32c0 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 4 set_mon_vals no callback set 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e39ffb640 10 monclient: dump: 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.677 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 10 monclient: shutdown 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 1 -- 172.21.15.138:0/1230356440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 msgr2=0x7f7e3c09e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 1 --2- 172.21.15.138:0/1230356440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c09e420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009870 tx=0x7f7e2c01c450 comp rx=0 tx=0).stop 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 1 -- 172.21.15.138:0/1230356440 shutdown_connections 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 1 --2- 172.21.15.138:0/1230356440 >> v2:172.21.15.138:3300/0 conn(0x7f7e3c09e960 0x7f7e3c09ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 1 --2- 172.21.15.138:0/1230356440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c09e420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 1 -- 172.21.15.138:0/1230356440 >> 172.21.15.138:0/1230356440 conn(0x7f7e3c063640 msgr2=0x7f7e3c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:21.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 1 -- 172.21.15.138:0/1230356440 shutdown_connections 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 1 -- 172.21.15.138:0/1230356440 wait complete. 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 10 monclient: build_initial_monmap 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.672+0000 7f7e414d6640 10 monclient: monmap: 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 librados: starting msgr at 2024-09-11T05:18:21.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 librados: starting objecter 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 Processor -- start 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 -- start start 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 librados: setting wanted keys 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 librados: calling monclient init 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient: init 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c09e960 0x7f7e3c1349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient(hunting): picked mon.noname-b con 0x7f7e3c09e960 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c134f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient(hunting): picked mon.noname-c con 0x7f7e3c0a32c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e3c1356f0 0x7f7e3c11d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3c1356f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:21.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c140170 con 0x7f7e3c1356f0 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3c06e280 con 0x7f7e3c09e960 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c1402f0 con 0x7f7e3c0a32c0 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient(hunting): authenticate will time out at 1585.112775s 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c134f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e3c1356f0 0x7f7e3c11d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c134f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36228/0 (socket says 172.21.15.138:36228) 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 1 -- 172.21.15.138:0/3613054387 learned_addr learned my addr 172.21.15.138:0/3613054387 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient(hunting): get_auth_request con 0x7f7e3c1356f0 auth_method 0 2024-09-11T05:18:21.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 10 monclient(hunting): get_auth_request con 0x7f7e3c0a32c0 auth_method 0 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3affd640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c09e960 0x7f7e3c1349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3affd640 10 monclient(hunting): get_auth_request con 0x7f7e3c09e960 auth_method 0 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient(hunting): handle_auth_done global_id 64506 payload 824 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 msgr2=0x7f7e3c134f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c134f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c09e960 msgr2=0x7f7e3c1349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e3c09e960 0x7f7e3c1349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 monclient: found mon.noname-a 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 -- 172.21.15.138:0/3613054387 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3c0a4e60 con 0x7f7e3c1356f0 2024-09-11T05:18:21.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3a7fc640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c0a32c0 0x7f7e3c134f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:21.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e3c1356f0 msgr2=0x7f7e3c11d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e3b7fe640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e3c1356f0 0x7f7e3c11d570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400c780 tx=0x7f7e3400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 client.?.objecter ms_handle_connect 0x7f7e3c1356f0 2024-09-11T05:18:21.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:21.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34016020 con 0x7f7e3c1356f0 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: dump: 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.689 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e3400ac60 con 0x7f7e3c1356f0 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 5 monclient: authenticate success, global_id 64506 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient: _renew_subs 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34012660 con 0x7f7e3c1356f0 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3c063b00 con 0x7f7e3c1356f0 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e397fa640 4 set_mon_vals no callback set 2024-09-11T05:18:21.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e237fe640 10 monclient: dump: 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 client.64506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient: _renew_subs 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:21.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3c120450 con 0x7f7e3c1356f0 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.676+0000 7f7e414d6640 1 librados: init done 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e414d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e414d6640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02d450 0x7f7e3c02d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e414d6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c11e5f0 con 0x7f7e3c02d450 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02d450 0x7f7e3c02d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 10 monclient: get_auth_request con 0x7f7e3c02d450 auth_method 0 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f7e340128c0 con 0x7f7e3c1356f0 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 10 monclient(hunting): handle_auth_done global_id 84179 payload 824 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 1 -- 172.21.15.138:0/3613054387 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e3c0a51f0 con 0x7f7e3c02d450 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 --2- 172.21.15.138:0/3613054387 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e2803ced0 0x7f7e2803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f7e3404ff20 con 0x7f7e3c1356f0 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 10 client.64506.objecter ms_dispatch 0x7f7e3c000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 3 client.64506.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 3 client.64506.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 20 client.64506.objecter dump_active .. 0 homeless 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3a7fc640 1 --2- 172.21.15.138:0/3613054387 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e2803ced0 0x7f7e2803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3a7fc640 10 monclient: get_auth_request con 0x7f7e2803ced0 auth_method 0 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3a7fc640 1 -- 172.21.15.138:0/3613054387 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e2803ced0 msgr2=0x7f7e2803f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3a7fc640 1 -- 172.21.15.138:0/3613054387 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e2803ced0 msgr2=0x7f7e2803f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:18:21.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3a7fc640 1 --2- 172.21.15.138:0/3613054387 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e2803ced0 0x7f7e2803f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3a7fc640 1 --2- 172.21.15.138:0/3613054387 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e2803ced0 0x7f7e2803f390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 10 client.64506.objecter ms_handle_connect 0x7f7e3c02d450 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02d450 msgr2=0x7f7e3c02d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 10 client.64506.objecter resend_mon_ops 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e3affd640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02d450 0x7f7e3c02d830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0ad000 tx=0x7f7e2c019250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c0bb070 con 0x7f7e3c02d450 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 10 monclient: dump: 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.694 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e2c0a4e60 con 0x7f7e3c02d450 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02d450 msgr2=0x7f7e3c02d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02d450 0x7f7e3c02d830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0ad000 tx=0x7f7e2c019250 comp rx=0 tx=0).stop 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02d450 msgr2=0x7f7e3c02d830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.680+0000 7f7e237fe640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02d450 0x7f7e3c02d830 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e414d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:18:21.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e414d6640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02dd70 0x7f7e3c02e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e414d6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7e3c073470 con 0x7f7e3c02dd70 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02dd70 0x7f7e3c02e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 10 monclient: get_auth_request con 0x7f7e3c02dd70 auth_method 0 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 10 monclient(hunting): handle_auth_done global_id 84182 payload 824 2024-09-11T05:18:21.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.688+0000 7f7e3affd640 1 -- 172.21.15.138:0/3613054387 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e3c11ecb0 con 0x7f7e3c02dd70 2024-09-11T05:18:21.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e3affd640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02dd70 msgr2=0x7f7e3c02e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 10 client.64506.objecter ms_handle_connect 0x7f7e3c02dd70 2024-09-11T05:18:21.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 10 client.64506.objecter resend_mon_ops 2024-09-11T05:18:21.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e3affd640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02dd70 0x7f7e3c02e180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c01c490 tx=0x7f7e2c009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c0b0040 con 0x7f7e3c02dd70 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 10 monclient: dump: 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f7e2c0b7660 con 0x7f7e3c02dd70 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02dd70 msgr2=0x7f7e3c02e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02dd70 0x7f7e3c02e180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c01c490 tx=0x7f7e2c009240 comp rx=0 tx=0).stop 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02dd70 msgr2=0x7f7e3c02e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e237fe640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7e3c02dd70 0x7f7e3c02e180 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c01c490 tx=0x7f7e2c009240 comp rx=0 tx=0).stop 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:18:21.699 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 40, 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 145462, 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.700 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.701 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:18:21.702 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:18:21.703 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.704 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.705 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:21.706 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.707 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.708 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.709 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.710 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:21.711 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:21.711 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:18:21.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.711 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.711 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:18:21.711 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 10 librados: watch_flush enter 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 10 librados: watch_flush exit 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 20 client.64506.objecter shutdown clearing up homeless session... 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 10 client.64506.objecter successfully canceled tick 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e2803ced0 msgr2=0x7f7e2803f390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 --2- 172.21.15.138:0/3613054387 >> v2:172.21.15.190:6800/1562782189 conn(0x7f7e2803ced0 0x7f7e2803f390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 10 monclient: shutdown 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e3c1356f0 msgr2=0x7f7e3c11d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 --2- 172.21.15.138:0/3613054387 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f7e3c1356f0 0x7f7e3c11d570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400c780 tx=0x7f7e3400cc40 comp rx=0 tx=0).stop 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e3b7fe640 1 -- 172.21.15.138:0/3613054387 reap_dead start 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e3b7fe640 1 -- 172.21.15.138:0/3613054387 reap_dead start 2024-09-11T05:18:21.713 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 shutdown_connections 2024-09-11T05:18:21.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 >> 172.21.15.138:0/3613054387 conn(0x7f7e3c063640 msgr2=0x7f7e3c13cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:21.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 shutdown_connections 2024-09-11T05:18:21.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 -- 172.21.15.138:0/3613054387 wait complete. 2024-09-11T05:18:21.714 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.692+0000 7f7e414d6640 1 librados: shutdown 2024-09-11T05:18:21.714 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:18:21.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:18:21.714 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:18:21.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.828+0000 7f9cc31ca640 10 monclient: get_monmap_and_config 2024-09-11T05:18:21.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.828+0000 7f9cc31ca640 10 monclient: build_initial_monmap 2024-09-11T05:18:21.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.828+0000 7f9cc31ca640 10 monclient: monmap: 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:21.833287+0000 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:21.833287+0000 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:21.829 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 1 Processor -- start 2024-09-11T05:18:21.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 1 -- start start 2024-09-11T05:18:21.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient: init 2024-09-11T05:18:21.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-11T05:18:21.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9cbc138690 addr v1:172.21.15.190:6789/0 2024-09-11T05:18:21.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cbc0a34f0 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:21.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9cbc13ed90 0x7f9cbc141180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient(hunting): picked mon.noname-c con 0x7f9cbc13ed90 addr v2:172.21.15.190:3300/0 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cbc0a24a0 con 0x7f9cbc0a34f0 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cbc0a2780 con 0x7f9cbc138690 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9cbc0a3160 con 0x7f9cbc13ed90 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc31ca640 10 monclient(hunting): authenticate will time out at 1585.269510s 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cbbfff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f9cbc0a34f0 0x7f9cbc13d670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38410/0 (socket says 172.21.15.138:38410) 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cbbfff640 1 -- 172.21.15.138:0/1208803202 learned_addr learned my addr 172.21.15.138:0/1208803202 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc1767640 1 --2- 172.21.15.138:0/1208803202 >> v2:172.21.15.190:3300/0 conn(0x7f9cbc13ed90 0x7f9cbc141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc1767640 10 monclient(hunting): get_auth_request con 0x7f9cbc13ed90 auth_method 0 2024-09-11T05:18:21.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc1767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc1767640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc1767640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc1767640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc1767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cc1767640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cbbfff640 1 -- 172.21.15.138:0/1208803202 >> v1:172.21.15.138:6789/0 conn(0x7f9cbc0a34f0 legacy=0x7f9cbc13d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.832+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567994907 0 0) 0x7f9cbc0a24a0 con 0x7f9cbc0a34f0 2024-09-11T05:18:21.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): _negotiate 2024-09-11T05:18:21.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): my global_id is 64509 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ca4003530 con 0x7f9cbc0a34f0 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1090240424 0 0) 0x7f9ca4003530 con 0x7f9cbc0a34f0 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 >> v2:172.21.15.190:3300/0 conn(0x7f9cbc13ed90 msgr2=0x7f9cbc141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 --2- 172.21.15.138:0/1208803202 >> v2:172.21.15.190:3300/0 conn(0x7f9cbc13ed90 0x7f9cbc141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 >> v1:172.21.15.190:6789/0 conn(0x7f9cbc138690 legacy=0x7f9cbc138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:18:21.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc0a2a60 con 0x7f9cbc0a34f0 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc31ca640 5 monclient: authenticate success, global_id 64509 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f9cb00019b0 con 0x7f9cbc0a34f0 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc31ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: dump: 2024-09-11T05:18:21.838 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.839 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 >> v1:172.21.15.138:6789/0 conn(0x7f9cbc0a34f0 legacy=0x7f9cbc13d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ca4003ae0 0x7f9ca4003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): picked mon.a con 0x7f9ca4003ae0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca4004810 0x7f9ca4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): picked mon.b con 0x7f9ca4004810 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9ca400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): picked mon.c con 0x7f9ca40053d0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca400c280 con 0x7f9ca4003ae0 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca400c430 con 0x7f9ca4004810 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca400c630 con 0x7f9ca40053d0 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cbb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:18:21.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc31ca640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9ca400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient(hunting): get_auth_request con 0x7f9ca40053d0 auth_method 0 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc0f66640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ca4003ae0 0x7f9ca4003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc0f66640 10 monclient(hunting): get_auth_request con 0x7f9ca4003ae0 auth_method 0 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc0f66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc0f66640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc0f66640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient(hunting): handle_auth_done global_id 64509 payload 293 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 1 -- 172.21.15.138:0/1208803202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca4004810 msgr2=0x7f9ca4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ca4004810 0x7f9ca4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 1 -- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ca4003ae0 msgr2=0x7f9ca4003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ca4003ae0 0x7f9ca4003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 1 monclient: found mon.c 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 1 -- 172.21.15.138:0/1208803202 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca4003530 con 0x7f9ca40053d0 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc0f66640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9ca4003ae0 0x7f9ca4003ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:18:21.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.836+0000 7f9cc1767640 1 -- 172.21.15.138:0/1208803202 reap_dead start 2024-09-11T05:18:21.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cc1767640 1 -- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 msgr2=0x7f9ca400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.850 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cc1767640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9ca400b9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009fc0 tx=0x7f9cb401cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a45b9e18e464f25c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4009c10 con 0x7f9ca40053d0 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 10 monclient: dump: 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.851 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb40a5cb0 con 0x7f9ca40053d0 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 1 -- 172.21.15.138:0/1208803202 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb40a5e60 con 0x7f9ca40053d0 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cc31ca640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cc31ca640 4 set_mon_vals no callback set 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.840+0000 7f9cbb7fe640 10 monclient: dump: 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.852 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient: shutdown 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:21.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 msgr2=0x7f9ca400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9ca400b9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009fc0 tx=0x7f9cb401cca0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- 172.21.15.138:0/1208803202 shutdown_connections 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 --2- 172.21.15.138:0/1208803202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9ca400b9d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- 172.21.15.138:0/1208803202 >> 172.21.15.138:0/1208803202 conn(0x7f9cbc063640 msgr2=0x7f9cbc063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- 172.21.15.138:0/1208803202 shutdown_connections 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- 172.21.15.138:0/1208803202 wait complete. 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient: build_initial_monmap 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient: monmap: 2024-09-11T05:18:21.854 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 librados: starting msgr at 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 librados: starting objecter 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 Processor -- start 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- start start 2024-09-11T05:18:21.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 librados: setting wanted keys 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 librados: calling monclient init 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient: init 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9cbc131f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca40053d0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cbc132440 0x7f9cbc1393f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient(hunting): picked mon.noname-a con 0x7f9cbc132440 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc139930 0x7f9cbc139dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient(hunting): picked mon.noname-b con 0x7f9cbc139930 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc138a70 con 0x7f9cbc132440 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc138bf0 con 0x7f9cbc139930 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f9cbc138d70 con 0x7f9ca40053d0 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc31ca640 10 monclient(hunting): authenticate will time out at 1585.281710s 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cbc132440 0x7f9cbc1393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9cbc131f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cbc132440 0x7f9cbc1393f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43496/0 (socket says 172.21.15.138:43496) 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9cbc131f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36258/0 (socket says 172.21.15.138:36258) 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 1 -- 172.21.15.138:0/2972044143 learned_addr learned my addr 172.21.15.138:0/2972044143 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 10 monclient(hunting): get_auth_request con 0x7f9cbc132440 auth_method 0 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient(hunting): get_auth_request con 0x7f9ca40053d0 auth_method 0 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc1767640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc139930 0x7f9cbc139dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient(hunting): handle_auth_done global_id 84185 payload 824 2024-09-11T05:18:21.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc139930 msgr2=0x7f9cbc139dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc139930 0x7f9cbc139dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cbc132440 msgr2=0x7f9cbc1393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cbc132440 0x7f9cbc1393f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient: _finish_hunting 0 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 1 monclient: found mon.noname-c 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cc0f66640 1 -- 172.21.15.138:0/2972044143 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc13c7d0 con 0x7f9ca40053d0 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.844+0000 7f9cbbfff640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9cbc132440 0x7f9cbc1393f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc0f66640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 msgr2=0x7f9cbc131f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc0f66640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9cbc131f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00e4c0 tx=0x7f9cac00e980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 client.?.objecter ms_handle_connect 0x7f9ca40053d0 2024-09-11T05:18:21.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac018020 con 0x7f9ca40053d0 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: dump: 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.860 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: _finish_auth 0 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: _check_auth_tickets 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 5 monclient: authenticate success, global_id 84185 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 10 monclient: _renew_subs 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cac005030 con 0x7f9ca40053d0 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cbc13cab0 con 0x7f9ca40053d0 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac005400 con 0x7f9ca40053d0 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 monclient: dump: 2024-09-11T05:18:21.861 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbaffd640 4 set_mon_vals no callback set 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 10 client.84185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 10 monclient: _renew_subs 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:21.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cbc136280 con 0x7f9ca40053d0 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 1 librados: init done 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13a300 0x7f9cbc13f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc138500 con 0x7f9cbc13a300 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f9cac013800 con 0x7f9ca40053d0 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13a300 0x7f9cbc13f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 1 --2- 172.21.15.138:0/2972044143 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9ca803cef0 0x7f9ca803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 10 monclient: get_auth_request con 0x7f9cbc13a300 auth_method 0 2024-09-11T05:18:21.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f9cac04f680 con 0x7f9ca40053d0 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 10 client.84185.objecter ms_dispatch 0x7f9cbc000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 3 client.84185.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 3 client.84185.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cb97fa640 20 client.84185.objecter dump_active .. 0 homeless 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc1767640 1 --2- 172.21.15.138:0/2972044143 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9ca803cef0 0x7f9ca803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cc1767640 10 monclient: get_auth_request con 0x7f9ca803cef0 auth_method 0 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_done global_id 94111 payload 824 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.848+0000 7f9cbbfff640 1 -- 172.21.15.138:0/2972044143 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cbc0a24a0 con 0x7f9cbc13a300 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 10 client.84185.objecter ms_handle_connect 0x7f9cbc13a300 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cbbfff640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13a300 msgr2=0x7f9cbc13f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 10 client.84185.objecter resend_mon_ops 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cbbfff640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13a300 0x7f9cbc13f270 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0004d90 tx=0x7f9cb000f5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 10 client.84185.objecter ms_handle_connect 0x7f9ca803cef0 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cc1767640 1 --2- 172.21.15.138:0/2972044143 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9ca803cef0 0x7f9ca803f3b0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0a39f0 tx=0x7f9cb4004ac0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0007e40 con 0x7f9cbc13a300 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 10 monclient: dump: 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.865 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9cb0018070 con 0x7f9cbc13a300 2024-09-11T05:18:21.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:21.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:21.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13a300 msgr2=0x7f9cbc13f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13a300 0x7f9cbc13f270 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0004d90 tx=0x7f9cb000f5a0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13a300 msgr2=0x7f9cbc13f270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:21.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.852+0000 7f9cb97fa640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13a300 0x7f9cbc13f270 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cc31ca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cc31ca640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13f7b0 0x7f9cbc13fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cc31ca640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc13aa30 con 0x7f9cbc13f7b0 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13f7b0 0x7f9cbc13fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 10 monclient: get_auth_request con 0x7f9cbc13f7b0 auth_method 0 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:21.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 10 monclient(hunting): handle_auth_done global_id 94114 payload 824 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 1 -- 172.21.15.138:0/2972044143 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9cbc06cca0 con 0x7f9cbc13f7b0 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13f7b0 msgr2=0x7f9cbc13fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cb97fa640 10 client.84185.objecter ms_handle_connect 0x7f9cbc13f7b0 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cb97fa640 10 client.84185.objecter resend_mon_ops 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.860+0000 7f9cbbfff640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13f7b0 0x7f9cbc13fbc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0004d90 tx=0x7f9cb00013d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0012bb0 con 0x7f9cbc13f7b0 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 10 monclient: dump: 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:21.870 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f9cb0012bb0 con 0x7f9cbc13f7b0 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13f7b0 msgr2=0x7f9cbc13fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13f7b0 0x7f9cbc13fbc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0004d90 tx=0x7f9cb00013d0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13f7b0 msgr2=0x7f9cbc13fbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cb97fa640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc13f7b0 0x7f9cbc13fbc0 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0004d90 tx=0x7f9cb00013d0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 40, 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 15811, 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:18:21.872 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.873 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.874 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:18:21.875 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:21.876 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.877 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:18:21.878 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:21.879 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.880 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:21.881 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.882 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:21.883 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:21.884 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:21.885 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:21.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:21.886 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:18:21.886 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:21.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 10 librados: watch_flush enter 2024-09-11T05:18:21.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 10 librados: watch_flush exit 2024-09-11T05:18:21.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 20 client.84185.objecter shutdown clearing up homeless session... 2024-09-11T05:18:21.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 10 client.84185.objecter successfully canceled tick 2024-09-11T05:18:21.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9ca803cef0 msgr2=0x7f9ca803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 --2- 172.21.15.138:0/2972044143 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9ca803cef0 0x7f9ca803f3b0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0a39f0 tx=0x7f9cb4004ac0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 10 monclient: shutdown 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 msgr2=0x7f9cbc131f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9cbc131f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00e4c0 tx=0x7f9cac00e980 comp rx=0 tx=0).stop 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc1767640 1 -- 172.21.15.138:0/2972044143 reap_dead start 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 shutdown_connections 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 --2- 172.21.15.138:0/2972044143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9ca40053d0 0x7f9cbc131f00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 >> 172.21.15.138:0/2972044143 conn(0x7f9cbc063640 msgr2=0x7f9cbc134ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 shutdown_connections 2024-09-11T05:18:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 -- 172.21.15.138:0/2972044143 wait complete. 2024-09-11T05:18:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:21.864+0000 7f9cc31ca640 1 librados: shutdown 2024-09-11T05:18:21.890 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:18:21.890 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:18:21.890 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 10 monclient: get_monmap_and_config 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 10 monclient: build_initial_monmap 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 10 monclient: monmap: 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:22.007849+0000 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:22.007849+0000 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:22.003 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:22.004 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:22.004 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:22.004 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:22.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 1 Processor -- start 2024-09-11T05:18:22.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 1 -- start start 2024-09-11T05:18:22.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 10 monclient: init 2024-09-11T05:18:22.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:22.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-11T05:18:22.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f40f406e0e0 0x7f40f409e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:22.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 10 monclient(hunting): picked mon.noname-a con 0x7f40f406e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:22.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.004+0000 7f40f988d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40f409e960 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:22.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f988d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40f409ed70 addr v1:172.21.15.190:6789/0 2024-09-11T05:18:22.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f988d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40f4136510 con 0x7f40f409e960 2024-09-11T05:18:22.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f988d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40f409f540 con 0x7f40f409ed70 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f988d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f988d640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f40f406e4c0 con 0x7f40f406e0e0 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f988d640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f40f406e0e0 0x7f40f409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f988d640 10 monclient(hunting): authenticate will time out at 1585.444143s 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f40f406e0e0 0x7f40f409e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43500/0 (socket says 172.21.15.138:43500) 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 1 -- 172.21.15.138:0/936812852 learned_addr learned my addr 172.21.15.138:0/936812852 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient(hunting): get_auth_request con 0x7f40f406e0e0 auth_method 0 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:22.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:22.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:22.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:22.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:22.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:22.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient(hunting): handle_auth_done global_id 64515 payload 274 2024-09-11T05:18:22.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 1 -- 172.21.15.138:0/936812852 >> v1:172.21.15.190:6789/0 conn(0x7f40f409ed70 legacy=0x7f40f4135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:22.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 1 -- 172.21.15.138:0/936812852 >> v1:172.21.15.138:6789/0 conn(0x7f40f409e960 legacy=0x7f40f41356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:22.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:18:22.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 1 monclient: found mon.noname-a 2024-09-11T05:18:22.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:22.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:22.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.008+0000 7f40f2ffd640 1 -- 172.21.15.138:0/936812852 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f409fb00 con 0x7f40f406e0e0 2024-09-11T05:18:22.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 -- 172.21.15.138:0/936812852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 msgr2=0x7f40f409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:22.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 --2- 172.21.15.138:0/936812852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f409e420 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f40e40098a0 tx=0x7f40e401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ecb6e7e0075c5f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:22.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 1 -- 172.21.15.138:0/936812852 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e40b0070 con 0x7f40f406e0e0 2024-09-11T05:18:22.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:22.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:22.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: dump: 2024-09-11T05:18:22.019 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:22.019 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: _finish_auth 0 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 1 -- 172.21.15.138:0/936812852 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e40ab600 con 0x7f40f406e0e0 2024-09-11T05:18:22.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 1 -- 172.21.15.138:0/936812852 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e40ab9d0 con 0x7f40f406e0e0 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f1ffb640 10 monclient: dump: 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:22.021 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 5 monclient: authenticate success, global_id 64515 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 4 set_mon_vals no callback set 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient: shutdown 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- 172.21.15.138:0/936812852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 msgr2=0x7f40f409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 --2- 172.21.15.138:0/936812852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f409e420 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f40e40098a0 tx=0x7f40e401c490 comp rx=0 tx=0).stop 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- 172.21.15.138:0/936812852 shutdown_connections 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 --2- 172.21.15.138:0/936812852 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f409e420 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- 172.21.15.138:0/936812852 >> 172.21.15.138:0/936812852 conn(0x7f40f4063640 msgr2=0x7f40f4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- 172.21.15.138:0/936812852 shutdown_connections 2024-09-11T05:18:22.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- 172.21.15.138:0/936812852 wait complete. 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient: build_initial_monmap 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient: monmap: 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:22.023 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 librados: starting msgr at 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 librados: starting objecter 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 Processor -- start 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- start start 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 librados: setting wanted keys 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 librados: calling monclient init 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient: init 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:22.024 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f41cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient(hunting): picked mon.noname-a con 0x7f40f406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40f409e960 0x7f40f41355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient(hunting): picked mon.noname-c con 0x7f40f409e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f409ed70 0x7f40f413c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient(hunting): picked mon.noname-b con 0x7f40f409ed70 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f40f4141590 con 0x7f40f406e0e0 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40f409f290 con 0x7f40f409ed70 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f40f4141710 con 0x7f40f409e960 2024-09-11T05:18:22.025 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f988d640 10 monclient(hunting): authenticate will time out at 1585.451212s 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f41cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f41cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43510/0 (socket says 172.21.15.138:43510) 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 -- 172.21.15.138:0/3434874561 learned_addr learned my addr 172.21.15.138:0/3434874561 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient(hunting): get_auth_request con 0x7f40f406e0e0 auth_method 0 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:22.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f27fc640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40f409e960 0x7f40f41355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f27fc640 10 monclient(hunting): get_auth_request con 0x7f40f409e960 auth_method 0 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f37fe640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f409ed70 0x7f40f413c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f37fe640 10 monclient(hunting): get_auth_request con 0x7f40f409ed70 auth_method 0 2024-09-11T05:18:22.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient(hunting): handle_auth_done global_id 64518 payload 824 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 -- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40f409e960 msgr2=0x7f40f41355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40f409e960 0x7f40f41355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 -- 172.21.15.138:0/3434874561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f409ed70 msgr2=0x7f40f413c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f409ed70 0x7f40f413c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 monclient: found mon.noname-a 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.012+0000 7f40f2ffd640 1 -- 172.21.15.138:0/3434874561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f411e9b0 con 0x7f40f406e0e0 2024-09-11T05:18:22.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f2ffd640 1 -- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 msgr2=0x7f40f41cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f2ffd640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f41cd7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f40e40ae040 tx=0x7f40e401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 client.?.objecter ms_handle_connect 0x7f40f406e0e0 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 1 -- 172.21.15.138:0/3434874561 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e40b8070 con 0x7f40f406e0e0 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: dump: 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:22.029 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: _finish_auth 0 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: _check_auth_tickets 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 1 -- 172.21.15.138:0/3434874561 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e40bc620 con 0x7f40f406e0e0 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 1 -- 172.21.15.138:0/3434874561 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e40bca10 con 0x7f40f406e0e0 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:22.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 monclient: dump: 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 5 monclient: authenticate success, global_id 64518 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 10 monclient: _renew_subs 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f17fa640 4 set_mon_vals no callback set 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f406fa30 con 0x7f40f406e0e0 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 10 client.64518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 10 monclient: _renew_subs 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f40735e0 con 0x7f40f406e0e0 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 1 librados: init done 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f409e420 con 0x7f40f406e0e0 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 1 -- 172.21.15.138:0/3434874561 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f40e40c5450 con 0x7f40f406e0e0 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 1 --2- 172.21.15.138:0/3434874561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f40e003ced0 0x7f40e003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 1 -- 172.21.15.138:0/3434874561 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f40e40eb580 con 0x7f40f406e0e0 2024-09-11T05:18:22.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 10 client.64518.objecter ms_dispatch 0x7f40f4000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:22.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 3 client.64518.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:22.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 3 client.64518.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:22.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40dbfff640 20 client.64518.objecter dump_active .. 0 homeless 2024-09-11T05:18:22.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f27fc640 1 --2- 172.21.15.138:0/3434874561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f40e003ced0 0x7f40e003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:22.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.016+0000 7f40f27fc640 10 monclient: get_auth_request con 0x7f40e003ced0 auth_method 0 2024-09-11T05:18:22.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.020+0000 7f40f27fc640 1 --2- 172.21.15.138:0/3434874561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f40e003ced0 0x7f40e003f390 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7f40f411ebc0 tx=0x7f40e8007e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:22.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.020+0000 7f40dbfff640 10 client.64518.objecter ms_handle_connect 0x7f40e003ced0 2024-09-11T05:18:22.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.024+0000 7f40dbfff640 1 -- 172.21.15.138:0/3434874561 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f40e4016110 con 0x7f40f406e0e0 2024-09-11T05:18:22.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.024+0000 7f40dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:22.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.024+0000 7f40dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:22.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.240+0000 7f40f988d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:18:22.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.240+0000 7f40f988d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:18:22.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.240+0000 7f40f988d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:22.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.240+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f40f409dea0 con 0x7f40f406e0e0 2024-09-11T05:18:22.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.240+0000 7f40dbfff640 1 -- 172.21.15.138:0/3434874561 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f40e40c5c90 con 0x7f40f406e0e0 2024-09-11T05:18:22.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.240+0000 7f40dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:18:22.239 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.240+0000 7f40dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:22.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 10 librados: watch_flush enter 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 10 librados: watch_flush exit 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 20 client.64518.objecter shutdown clearing up homeless session... 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 10 client.64518.objecter successfully canceled tick 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f40e003ced0 msgr2=0x7f40e003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 --2- 172.21.15.138:0/3434874561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f40e003ced0 0x7f40e003f390 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7f40f411ebc0 tx=0x7f40e8007e40 comp rx=0 tx=0).stop 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 10 monclient: shutdown 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 msgr2=0x7f40f41cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f41cd7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f40e40ae040 tx=0x7f40e401acd0 comp rx=0 tx=0).stop 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 shutdown_connections 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 --2- 172.21.15.138:0/3434874561 >> v2:172.21.15.190:6800/1562782189 conn(0x7f40e003ced0 0x7f40e003f390 unknown :-1 s=CLOSED pgs=3435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40f409ed70 0x7f40f413c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:22.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40f409e960 0x7f40f41355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:22.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 --2- 172.21.15.138:0/3434874561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40f406e0e0 0x7f40f41cd7a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:22.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 >> 172.21.15.138:0/3434874561 conn(0x7f40f4063640 msgr2=0x7f40f4136910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:22.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 shutdown_connections 2024-09-11T05:18:22.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 -- 172.21.15.138:0/3434874561 wait complete. 2024-09-11T05:18:22.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:22.244+0000 7f40f988d640 1 librados: shutdown 2024-09-11T05:18:22.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.248+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.248+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=10,kv=9,logm=63} crc {auth=3611874956,config=692619927,health=2345169553,kv=534670036,logm=1701162606}) 2024-09-11T05:18:22.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.248+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=10,kv=9,logm=64} crc {auth=136239756,config=692619927,health=1922949966,kv=2751192215,logm=1651104610}) 2024-09-11T05:18:22.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.248+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.248+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=10,kv=9,logm=63} crc {auth=3611874956,config=692619927,health=2345169553,kv=534670036,logm=1701162606}) 2024-09-11T05:18:22.246 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.248+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=1173943601,config=692619927,health=3623406937,kv=310889196,logm=114632242}) 2024-09-11T05:18:22.249 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.249 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1375720797}) 2024-09-11T05:18:22.249 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1658189719}) 2024-09-11T05:18:22.249 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.249 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1375720797}) 2024-09-11T05:18:22.249 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520571212}) 2024-09-11T05:18:22.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622841682}) 2024-09-11T05:18:22.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273444969}) 2024-09-11T05:18:22.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622841682}) 2024-09-11T05:18:22.252 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.252+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697804310}) 2024-09-11T05:18:22.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.256+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.256+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121312238}) 2024-09-11T05:18:22.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.256+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3338301558}) 2024-09-11T05:18:22.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.256+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.256+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121312238}) 2024-09-11T05:18:22.254 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.256+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768513996}) 2024-09-11T05:18:22.258 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:18:22.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496763297}) 2024-09-11T05:18:22.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1119972478}) 2024-09-11T05:18:22.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496763297}) 2024-09-11T05:18:22.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2851531167}) 2024-09-11T05:18:22.899 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.899 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182964752}) 2024-09-11T05:18:22.899 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4176148188}) 2024-09-11T05:18:22.899 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.899 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182964752}) 2024-09-11T05:18:22.899 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.900+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4265980700}) 2024-09-11T05:18:22.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2803591129}) 2024-09-11T05:18:22.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=631072971}) 2024-09-11T05:18:22.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.901 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2803591129}) 2024-09-11T05:18:22.902 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149281867}) 2024-09-11T05:18:22.904 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.904 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661224139}) 2024-09-11T05:18:22.904 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686305890}) 2024-09-11T05:18:22.904 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.904 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661224139}) 2024-09-11T05:18:22.905 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.904+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748569961}) 2024-09-11T05:18:22.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341827540}) 2024-09-11T05:18:22.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3610090770}) 2024-09-11T05:18:22.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341827540}) 2024-09-11T05:18:22.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080218235}) 2024-09-11T05:18:22.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212179351}) 2024-09-11T05:18:22.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=806210052}) 2024-09-11T05:18:22.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4212179351}) 2024-09-11T05:18:22.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.908+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659960207}) 2024-09-11T05:18:22.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.912+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.912+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310198581}) 2024-09-11T05:18:22.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.912+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=35525802}) 2024-09-11T05:18:22.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.912+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.912+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310198581}) 2024-09-11T05:18:22.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.912+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3282966643}) 2024-09-11T05:18:22.912 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.912 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371140996}) 2024-09-11T05:18:22.913 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069730454}) 2024-09-11T05:18:22.913 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.913 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371140996}) 2024-09-11T05:18:22.913 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3773859467}) 2024-09-11T05:18:22.915 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.915 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743443025}) 2024-09-11T05:18:22.915 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863861271}) 2024-09-11T05:18:22.915 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.915 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743443025}) 2024-09-11T05:18:22.915 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.916+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2237056322}) 2024-09-11T05:18:22.917 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.917 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186784974}) 2024-09-11T05:18:22.917 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1773423825}) 2024-09-11T05:18:22.917 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.917 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186784974}) 2024-09-11T05:18:22.917 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771143665}) 2024-09-11T05:18:22.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155264464}) 2024-09-11T05:18:22.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722870196}) 2024-09-11T05:18:22.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155264464}) 2024-09-11T05:18:22.920 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.920+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279522975}) 2024-09-11T05:18:22.921 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.921 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3823916181}) 2024-09-11T05:18:22.922 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2915749433}) 2024-09-11T05:18:22.922 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.922 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3823916181}) 2024-09-11T05:18:22.922 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=495602554}) 2024-09-11T05:18:22.924 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.924 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159313516}) 2024-09-11T05:18:22.924 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449231038}) 2024-09-11T05:18:22.924 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.924 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159313516}) 2024-09-11T05:18:22.924 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.924+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2321495375}) 2024-09-11T05:18:22.926 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.926 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175147552}) 2024-09-11T05:18:22.926 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646342070}) 2024-09-11T05:18:22.926 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.926 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175147552}) 2024-09-11T05:18:22.926 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2810553776}) 2024-09-11T05:18:22.928 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.928 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463393978}) 2024-09-11T05:18:22.929 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2051671527}) 2024-09-11T05:18:22.929 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.928+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.929 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.932+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463393978}) 2024-09-11T05:18:22.929 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.932+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622336617}) 2024-09-11T05:18:22.930 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.932+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.930 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.932+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164722154}) 2024-09-11T05:18:22.931 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.932+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=297603276}) 2024-09-11T05:18:22.931 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.932+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.931 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.932+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4164722154}) 2024-09-11T05:18:22.931 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.932+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2033784110}) 2024-09-11T05:18:22.933 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.933 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937519823}) 2024-09-11T05:18:22.933 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2587855252}) 2024-09-11T05:18:22.933 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.933 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937519823}) 2024-09-11T05:18:22.933 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2792188460}) 2024-09-11T05:18:22.935 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.935 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367548982}) 2024-09-11T05:18:22.935 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931502264}) 2024-09-11T05:18:22.935 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.935 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367548982}) 2024-09-11T05:18:22.936 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.936+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2262107203}) 2024-09-11T05:18:22.990 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.990 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467893789}) 2024-09-11T05:18:22.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1928811712}) 2024-09-11T05:18:22.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:22.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467893789}) 2024-09-11T05:18:22.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:22.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266334868}) 2024-09-11T05:18:23.070 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.072+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.070 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.072+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283627462}) 2024-09-11T05:18:23.070 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.072+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1450707289}) 2024-09-11T05:18:23.071 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.072+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.071 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.072+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283627462}) 2024-09-11T05:18:23.071 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.072+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=367060356}) 2024-09-11T05:18:23.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.080+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.080+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736995756}) 2024-09-11T05:18:23.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.080+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848567358}) 2024-09-11T05:18:23.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.080+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.080+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736995756}) 2024-09-11T05:18:23.080 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.080+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174693837}) 2024-09-11T05:18:23.099 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.099 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=620222818}) 2024-09-11T05:18:23.099 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2165824803}) 2024-09-11T05:18:23.099 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.099 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=620222818}) 2024-09-11T05:18:23.099 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691054133}) 2024-09-11T05:18:23.259 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:18:23.259 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:18:23.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient: get_monmap_and_config 2024-09-11T05:18:23.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient: build_initial_monmap 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient: monmap: 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:23.391453+0000 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:23.391453+0000 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:23.387 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 1 Processor -- start 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 1 -- start start 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient: init 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa09406e0e0 0x7fa094136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient(hunting): picked mon.noname-a con 0x7fa09406e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa094136170 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa094136c60 0x7fa0940a2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient(hunting): picked mon.noname-b con 0x7fa094136c60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0940a3270 con 0x7fa094136170 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fa09406e4c0 con 0x7fa09406e0e0 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa0940a0af0 con 0x7fa094136c60 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa099c7a640 10 monclient(hunting): authenticate will time out at 1586.827515s 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fa09406e0e0 0x7fa094136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa092ffd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fa094136170 0x7fa0940a1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38420/0 (socket says 172.21.15.138:38420) 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa092ffd640 1 -- 172.21.15.138:0/2374469329 learned_addr learned my addr 172.21.15.138:0/2374469329 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:18:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa093fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa094136c60 0x7fa0940a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 10 monclient(hunting): get_auth_request con 0x7fa09406e0e0 auth_method 0 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa093fff640 10 monclient(hunting): get_auth_request con 0x7fa094136c60 auth_method 0 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa093fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa093fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa093fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa093fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 10 monclient(hunting): handle_auth_done global_id 64521 payload 274 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 1 -- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa094136c60 msgr2=0x7fa0940a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.388+0000 7fa0937fe640 1 --2- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa094136c60 0x7fa0940a2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.392+0000 7fa0937fe640 1 -- 172.21.15.138:0/2374469329 >> v1:172.21.15.138:6789/0 conn(0x7fa094136170 legacy=0x7fa0940a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.392+0000 7fa0937fe640 10 monclient: _finish_hunting 0 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.392+0000 7fa0937fe640 1 monclient: found mon.noname-a 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.392+0000 7fa0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.392+0000 7fa0937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.392+0000 7fa0937fe640 1 -- 172.21.15.138:0/2374469329 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0940a2f90 con 0x7fa09406e0e0 2024-09-11T05:18:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.392+0000 7fa093fff640 1 --2- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa094136c60 0x7fa0940a2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 -- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa09406e0e0 msgr2=0x7fa094136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 --2- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa09406e0e0 0x7fa094136720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa0880097f0 tx=0x7fa08801c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8131ed35e50a22d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 1 -- 172.21.15.138:0/2374469329 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0880b1070 con 0x7fa09406e0e0 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: dump: 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.402 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: _finish_auth 0 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: _check_auth_tickets 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 5 monclient: authenticate success, global_id 64521 2024-09-11T05:18:23.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 1 -- 172.21.15.138:0/2374469329 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0880a4d80 con 0x7fa09406e0e0 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 4 set_mon_vals no callback set 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 1 -- 172.21.15.138:0/2374469329 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0880ac660 con 0x7fa09406e0e0 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0927fc640 10 monclient: dump: 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.404 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient: shutdown 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa09406e0e0 msgr2=0x7fa094136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 --2- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa09406e0e0 0x7fa094136720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa0880097f0 tx=0x7fa08801c120 comp rx=0 tx=0).stop 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- 172.21.15.138:0/2374469329 shutdown_connections 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 --2- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa094136c60 0x7fa0940a2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 --2- 172.21.15.138:0/2374469329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa09406e0e0 0x7fa094136720 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- 172.21.15.138:0/2374469329 >> 172.21.15.138:0/2374469329 conn(0x7fa094063640 msgr2=0x7fa094063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- 172.21.15.138:0/2374469329 shutdown_connections 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- 172.21.15.138:0/2374469329 wait complete. 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient: build_initial_monmap 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient: monmap: 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:23.406 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 librados: starting msgr at 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 librados: starting objecter 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 Processor -- start 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- start start 2024-09-11T05:18:23.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 librados: setting wanted keys 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 librados: calling monclient init 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient: init 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa09406e0e0 0x7fa094070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient(hunting): picked mon.noname-c con 0x7fa09406e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094136170 0x7fa094071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient(hunting): picked mon.noname-b con 0x7fa094136170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa094136c60 0x7fa094071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient(hunting): picked mon.noname-a con 0x7fa094136c60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fa094140f20 con 0x7fa094136c60 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa0940a2780 con 0x7fa094136170 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fa0941410a0 con 0x7fa09406e0e0 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa099c7a640 10 monclient(hunting): authenticate will time out at 1586.834658s 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa09406e0e0 0x7fa094070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa09406e0e0 0x7fa094070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36280/0 (socket says 172.21.15.138:36280) 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 -- 172.21.15.138:0/197637432 learned_addr learned my addr 172.21.15.138:0/197637432 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient(hunting): get_auth_request con 0x7fa09406e0e0 auth_method 0 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa094136c60 0x7fa094071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 10 monclient(hunting): get_auth_request con 0x7fa094136c60 auth_method 0 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa092ffd640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094136170 0x7fa094071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa092ffd640 10 monclient(hunting): get_auth_request con 0x7fa094136170 auth_method 0 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa092ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa092ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient(hunting): handle_auth_done global_id 84194 payload 824 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 -- 172.21.15.138:0/197637432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094136170 msgr2=0x7fa094071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094136170 0x7fa094071040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 -- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa094136c60 msgr2=0x7fa094071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa094136c60 0x7fa094071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient: _finish_hunting 0 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 monclient: found mon.noname-c 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa0937fe640 1 -- 172.21.15.138:0/197637432 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa094074120 con 0x7fa09406e0e0 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.396+0000 7fa093fff640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa094136c60 0x7fa094071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa0937fe640 1 -- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa09406e0e0 msgr2=0x7fa094070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa0937fe640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa09406e0e0 0x7fa094070b00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa0880097f0 tx=0x7fa088018f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 client.?.objecter ms_handle_connect 0x7fa09406e0e0 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 1 -- 172.21.15.138:0/197637432 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0880b9070 con 0x7fa09406e0e0 2024-09-11T05:18:23.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: dump: 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.413 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: _finish_auth 0 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 5 monclient: authenticate success, global_id 84194 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 10 monclient: _renew_subs 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 1 -- 172.21.15.138:0/197637432 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0880b5410 con 0x7fa09406e0e0 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa094137320 con 0x7fa09406e0e0 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 1 -- 172.21.15.138:0/197637432 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0880b1040 con 0x7fa09406e0e0 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 monclient: dump: 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.415 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 10 client.84194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 10 monclient: _renew_subs 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa091ffb640 4 set_mon_vals no callback set 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa094073e40 con 0x7fa09406e0e0 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 1 librados: init done 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa094136720 con 0x7fa09406e0e0 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 1 -- 172.21.15.138:0/197637432 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fa0880b4050 con 0x7fa09406e0e0 2024-09-11T05:18:23.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 1 --2- 172.21.15.138:0/197637432 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa08003cea0 0x7fa08003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 1 -- 172.21.15.138:0/197637432 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7fa0880e92e0 con 0x7fa09406e0e0 2024-09-11T05:18:23.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 10 client.84194.objecter ms_dispatch 0x7fa094000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:23.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 3 client.84194.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:23.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 3 client.84194.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:23.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa07bfff640 20 client.84194.objecter dump_active .. 0 homeless 2024-09-11T05:18:23.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa092ffd640 1 --2- 172.21.15.138:0/197637432 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa08003cea0 0x7fa08003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.400+0000 7fa092ffd640 10 monclient: get_auth_request con 0x7fa08003cea0 auth_method 0 2024-09-11T05:18:23.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.404+0000 7fa092ffd640 1 --2- 172.21.15.138:0/197637432 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa08003cea0 0x7fa08003f360 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7fa094074330 tx=0x7fa084009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.404+0000 7fa07bfff640 10 client.84194.objecter ms_handle_connect 0x7fa08003cea0 2024-09-11T05:18:23.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.404+0000 7fa07bfff640 1 -- 172.21.15.138:0/197637432 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fa0940a3270 con 0x7fa09406e0e0 2024-09-11T05:18:23.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.404+0000 7fa07bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:23.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.404+0000 7fa07bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:23.503 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.504+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.503 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.504+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425894323}) 2024-09-11T05:18:23.503 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.504+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174858172}) 2024-09-11T05:18:23.503 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.504+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.503 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.504+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425894323}) 2024-09-11T05:18:23.504 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.504+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574028915}) 2024-09-11T05:18:23.505 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.505 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051675149}) 2024-09-11T05:18:23.505 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40992719}) 2024-09-11T05:18:23.505 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.505 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051675149}) 2024-09-11T05:18:23.505 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188298028}) 2024-09-11T05:18:23.507 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.507 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285322082}) 2024-09-11T05:18:23.507 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=864135}) 2024-09-11T05:18:23.507 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.507 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285322082}) 2024-09-11T05:18:23.507 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.508+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712908329}) 2024-09-11T05:18:23.509 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.509 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609868535}) 2024-09-11T05:18:23.509 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497555167}) 2024-09-11T05:18:23.509 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.509 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1609868535}) 2024-09-11T05:18:23.509 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2398100274}) 2024-09-11T05:18:23.511 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.511 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003058142}) 2024-09-11T05:18:23.511 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945224531}) 2024-09-11T05:18:23.511 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.511 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003058142}) 2024-09-11T05:18:23.511 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.512+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611460002}) 2024-09-11T05:18:23.513 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.513 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928442509}) 2024-09-11T05:18:23.513 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58042894}) 2024-09-11T05:18:23.513 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.513 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928442509}) 2024-09-11T05:18:23.513 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351210226}) 2024-09-11T05:18:23.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242605959}) 2024-09-11T05:18:23.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1930725291}) 2024-09-11T05:18:23.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242605959}) 2024-09-11T05:18:23.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.516+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166070028}) 2024-09-11T05:18:23.516 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.517 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674345416}) 2024-09-11T05:18:23.517 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266887874}) 2024-09-11T05:18:23.517 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.517 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674345416}) 2024-09-11T05:18:23.517 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4187512009}) 2024-09-11T05:18:23.518 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.518 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265046410}) 2024-09-11T05:18:23.519 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3559438343}) 2024-09-11T05:18:23.519 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.519 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265046410}) 2024-09-11T05:18:23.519 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.520+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=773386568}) 2024-09-11T05:18:23.520 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.520 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356040397}) 2024-09-11T05:18:23.520 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628858587}) 2024-09-11T05:18:23.520 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.521 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356040397}) 2024-09-11T05:18:23.521 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2863641933}) 2024-09-11T05:18:23.522 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.522 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561628246}) 2024-09-11T05:18:23.522 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608034682}) 2024-09-11T05:18:23.522 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.522 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561628246}) 2024-09-11T05:18:23.522 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883102098}) 2024-09-11T05:18:23.524 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.524 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045681427}) 2024-09-11T05:18:23.524 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=83086663}) 2024-09-11T05:18:23.524 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.524 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045681427}) 2024-09-11T05:18:23.524 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.524+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575427897}) 2024-09-11T05:18:23.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781379415}) 2024-09-11T05:18:23.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1348490156}) 2024-09-11T05:18:23.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781379415}) 2024-09-11T05:18:23.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2417092943}) 2024-09-11T05:18:23.527 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661042604}) 2024-09-11T05:18:23.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953964054}) 2024-09-11T05:18:23.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661042604}) 2024-09-11T05:18:23.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.528+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3640125274}) 2024-09-11T05:18:23.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505222258}) 2024-09-11T05:18:23.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860727407}) 2024-09-11T05:18:23.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505222258}) 2024-09-11T05:18:23.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810285603}) 2024-09-11T05:18:23.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3072666247}) 2024-09-11T05:18:23.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345333572}) 2024-09-11T05:18:23.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:23.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3072666247}) 2024-09-11T05:18:23.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.532+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156152123}) 2024-09-11T05:18:23.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.652+0000 7fa099c7a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:18:23.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.652+0000 7fa099c7a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:23.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.652+0000 7fa099c7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:23.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.652+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa09406ccd0 con 0x7fa09406e0e0 2024-09-11T05:18:23.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa07bfff640 1 -- 172.21.15.138:0/197637432 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fa0880e9080 con 0x7fa09406e0e0 2024-09-11T05:18:23.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa07bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:23.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa07bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:23.655 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:18:23.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 10 librados: watch_flush enter 2024-09-11T05:18:23.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 10 librados: watch_flush exit 2024-09-11T05:18:23.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 20 client.84194.objecter shutdown clearing up homeless session... 2024-09-11T05:18:23.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 10 client.84194.objecter successfully canceled tick 2024-09-11T05:18:23.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa08003cea0 msgr2=0x7fa08003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 --2- 172.21.15.138:0/197637432 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa08003cea0 0x7fa08003f360 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7fa094074330 tx=0x7fa084009040 comp rx=0 tx=0).stop 2024-09-11T05:18:23.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 10 monclient: shutdown 2024-09-11T05:18:23.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:23.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa09406e0e0 msgr2=0x7fa094070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa09406e0e0 0x7fa094070b00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa0880097f0 tx=0x7fa088018f20 comp rx=0 tx=0).stop 2024-09-11T05:18:23.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 shutdown_connections 2024-09-11T05:18:23.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 --2- 172.21.15.138:0/197637432 >> v2:172.21.15.190:6800/1562782189 conn(0x7fa08003cea0 0x7fa08003f360 unknown :-1 s=CLOSED pgs=3436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fa094136c60 0x7fa094071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa094136170 0x7fa094071040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 --2- 172.21.15.138:0/197637432 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fa09406e0e0 0x7fa094070b00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 >> 172.21.15.138:0/197637432 conn(0x7fa094063640 msgr2=0x7fa09412c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:23.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 shutdown_connections 2024-09-11T05:18:23.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 -- 172.21.15.138:0/197637432 wait complete. 2024-09-11T05:18:23.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.656+0000 7fa099c7a640 1 librados: shutdown 2024-09-11T05:18:23.671 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:18:23.672 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:18:23.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:18:23.672 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:18:23.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.788+0000 7f3a8a1c9640 10 monclient: get_monmap_and_config 2024-09-11T05:18:23.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.788+0000 7f3a8a1c9640 10 monclient: build_initial_monmap 2024-09-11T05:18:23.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.788+0000 7f3a8a1c9640 10 monclient: monmap: 2024-09-11T05:18:23.788 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:23.793380+0000 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:23.793380+0000 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:23.789 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 1 Processor -- start 2024-09-11T05:18:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 1 -- start start 2024-09-11T05:18:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient: init 2024-09-11T05:18:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-11T05:18:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a840a32c0 addr v1:172.21.15.190:6789/0 2024-09-11T05:18:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3a8409eb30 0x7f3a8413a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient(hunting): picked mon.noname-a con 0x7f3a8409eb30 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a8409ef50 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a840a1b50 con 0x7f3a8409ef50 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a8409f560 con 0x7f3a840a32c0 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f3a8409fe70 con 0x7f3a8409eb30 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a8a1c9640 10 monclient(hunting): authenticate will time out at 1587.229451s 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3a8409eb30 0x7f3a8413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3a8409eb30 0x7f3a8413a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43546/0 (socket says 172.21.15.138:43546) 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a83fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f3a8409ef50 0x7f3a8413ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38424/0 (socket says 172.21.15.138:38424) 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 1 -- 172.21.15.138:0/2887329962 learned_addr learned my addr 172.21.15.138:0/2887329962 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient(hunting): get_auth_request con 0x7f3a8409eb30 auth_method 0 2024-09-11T05:18:23.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient(hunting): handle_auth_done global_id 64527 payload 274 2024-09-11T05:18:23.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 1 -- 172.21.15.138:0/2887329962 >> v1:172.21.15.190:6789/0 conn(0x7f3a840a32c0 legacy=0x7f3a8409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:23.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 1 -- 172.21.15.138:0/2887329962 >> v1:172.21.15.138:6789/0 conn(0x7f3a8409ef50 legacy=0x7f3a8413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:23.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:18:23.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 1 monclient: found mon.noname-a 2024-09-11T05:18:23.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:23.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.792+0000 7f3a82ffd640 1 -- 172.21.15.138:0/2887329962 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8409f840 con 0x7f3a8409eb30 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a82ffd640 1 -- 172.21.15.138:0/2887329962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409eb30 msgr2=0x7f3a8413a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a82ffd640 1 --2- 172.21.15.138:0/2887329962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409eb30 0x7f3a8413a930 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400bb10 tx=0x7f3a7401eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28db05c7e345b105 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 1 -- 172.21.15.138:0/2887329962 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a740b4070 con 0x7f3a8409eb30 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: dump: 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.802 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: _finish_auth 0 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: _check_auth_tickets 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 1 -- 172.21.15.138:0/2887329962 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a740a7920 con 0x7f3a8409eb30 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 5 monclient: authenticate success, global_id 64527 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 1 -- 172.21.15.138:0/2887329962 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a740a7cf0 con 0x7f3a8409eb30 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:23.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 4 set_mon_vals no callback set 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a827fc640 10 monclient: dump: 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 10 monclient: shutdown 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/2887329962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409eb30 msgr2=0x7f3a8413a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 1 --2- 172.21.15.138:0/2887329962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409eb30 0x7f3a8413a930 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400bb10 tx=0x7f3a7401eaf0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/2887329962 shutdown_connections 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 1 --2- 172.21.15.138:0/2887329962 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409eb30 0x7f3a8413a930 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/2887329962 >> 172.21.15.138:0/2887329962 conn(0x7f3a84063640 msgr2=0x7f3a84063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/2887329962 shutdown_connections 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/2887329962 wait complete. 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 10 monclient: build_initial_monmap 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.796+0000 7f3a8a1c9640 10 monclient: monmap: 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:23.805 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 librados: starting msgr at 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 librados: starting objecter 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 Processor -- start 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 -- start start 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 librados: setting wanted keys 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 librados: calling monclient init 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient: init 2024-09-11T05:18:23.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8409eb30 0x7f3a841d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient(hunting): picked mon.noname-b con 0x7f3a8409eb30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409ef50 0x7f3a84135a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient(hunting): picked mon.noname-a con 0x7f3a8409ef50 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3a840a32c0 0x7f3a84135f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient(hunting): picked mon.noname-c con 0x7f3a840a32c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3a840a2b40 con 0x7f3a8409ef50 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3a840a29c0 con 0x7f3a8409eb30 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3a840a2cc0 con 0x7f3a840a32c0 2024-09-11T05:18:23.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient(hunting): authenticate will time out at 1587.236765s 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409ef50 0x7f3a84135a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409ef50 0x7f3a84135a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43548/0 (socket says 172.21.15.138:43548) 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 -- 172.21.15.138:0/376970292 learned_addr learned my addr 172.21.15.138:0/376970292 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient(hunting): get_auth_request con 0x7f3a8409ef50 auth_method 0 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a83fff640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3a840a32c0 0x7f3a84135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a83fff640 10 monclient(hunting): get_auth_request con 0x7f3a840a32c0 auth_method 0 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a83fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a837fe640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8409eb30 0x7f3a841d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a837fe640 10 monclient(hunting): get_auth_request con 0x7f3a8409eb30 auth_method 0 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient(hunting): handle_auth_done global_id 64530 payload 824 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3a840a32c0 msgr2=0x7f3a84135f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3a840a32c0 0x7f3a84135f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8409eb30 msgr2=0x7f3a841d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a8409eb30 0x7f3a841d06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 monclient: found mon.noname-a 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 -- 172.21.15.138:0/376970292 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a84123100 con 0x7f3a8409ef50 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409ef50 msgr2=0x7f3a84135a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a82ffd640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409ef50 0x7f3a84135a20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b2c0 tx=0x7f3a7401d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 client.?.objecter ms_handle_connect 0x7f3a8409ef50 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a740bf070 con 0x7f3a8409ef50 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: dump: 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.811 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: _finish_auth 0 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a740afba0 con 0x7f3a8409ef50 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 5 monclient: authenticate success, global_id 64530 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient: _renew_subs 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a8413d080 con 0x7f3a8409ef50 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a740b4070 con 0x7f3a8409ef50 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a7bfff640 10 monclient: dump: 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a81ffb640 4 set_mon_vals no callback set 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 client.64530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:23.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient: _renew_subs 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a8413d950 con 0x7f3a8409ef50 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.800+0000 7f3a8a1c9640 1 librados: init done 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a8a1c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a8a1c9640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413bdd0 0x7f3a8402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a8a1c9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8409e110 con 0x7f3a8413bdd0 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3a7400e040 con 0x7f3a8409ef50 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413bdd0 0x7f3a8402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 10 monclient: get_auth_request con 0x7f3a8413bdd0 auth_method 0 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 --2- 172.21.15.138:0/376970292 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3a6403cea0 0x7f3a6403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 10 monclient(hunting): handle_auth_done global_id 64533 payload 824 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 1 -- 172.21.15.138:0/376970292 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a84122c40 con 0x7f3a8413bdd0 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f3a740e9af0 con 0x7f3a8409ef50 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 10 client.64530.objecter ms_dispatch 0x7f3a84000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 3 client.64530.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 3 client.64530.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 20 client.64530.objecter dump_active .. 0 homeless 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a83fff640 1 --2- 172.21.15.138:0/376970292 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3a6403cea0 0x7f3a6403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a83fff640 10 monclient: get_auth_request con 0x7f3a6403cea0 auth_method 0 2024-09-11T05:18:23.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 10 client.64530.objecter ms_handle_connect 0x7f3a8413bdd0 2024-09-11T05:18:23.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 10 client.64530.objecter resend_mon_ops 2024-09-11T05:18:23.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413bdd0 msgr2=0x7f3a8402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a837fe640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413bdd0 0x7f3a8402d450 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00e450 tx=0x7f3a6c00e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c007450 con 0x7f3a8413bdd0 2024-09-11T05:18:23.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 10 monclient: dump: 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a6c007450 con 0x7f3a8413bdd0 2024-09-11T05:18:23.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413bdd0 msgr2=0x7f3a8402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413bdd0 0x7f3a8402d450 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00e450 tx=0x7f3a6c00e910 comp rx=0 tx=0).stop 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413bdd0 msgr2=0x7f3a8402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.804+0000 7f3a7bfff640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413bdd0 0x7f3a8402d450 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00e450 tx=0x7f3a6c00e910 comp rx=0 tx=0).stop 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.808+0000 7f3a7bfff640 10 client.64530.objecter ms_handle_connect 0x7f3a6403cea0 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.808+0000 7f3a83fff640 1 --2- 172.21.15.138:0/376970292 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3a6403cea0 0x7f3a6403f360 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0078c0 tx=0x7f3a7c004f40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a8a1c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a8a1c9640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413c1b0 0x7f3a8402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a8a1c9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8409e420 con 0x7f3a8413c1b0 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413c1b0 0x7f3a8402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 10 monclient: get_auth_request con 0x7f3a8413c1b0 auth_method 0 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 10 monclient(hunting): handle_auth_done global_id 64536 payload 824 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.812+0000 7f3a837fe640 1 -- 172.21.15.138:0/376970292 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a84138020 con 0x7f3a8413c1b0 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a837fe640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413c1b0 msgr2=0x7f3a8402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 10 client.64530.objecter ms_handle_connect 0x7f3a8413c1b0 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 10 client.64530.objecter resend_mon_ops 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a837fe640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413c1b0 0x7f3a8402d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00e450 tx=0x7f3a6c003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c01eb70 con 0x7f3a8413c1b0 2024-09-11T05:18:23.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 10 monclient: dump: 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f3a6c028430 con 0x7f3a8413c1b0 2024-09-11T05:18:23.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413c1b0 msgr2=0x7f3a8402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413c1b0 0x7f3a8402d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00e450 tx=0x7f3a6c003770 comp rx=0 tx=0).stop 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413c1b0 msgr2=0x7f3a8402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a7bfff640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8413c1b0 0x7f3a8402d990 secure :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00e450 tx=0x7f3a6c003770 comp rx=0 tx=0).stop 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:18:23.823 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 40, 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 407446, 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:23.824 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:23.825 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:18:23.826 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:18:23.827 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.828 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:23.829 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:23.830 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.831 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:18:23.832 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:23.833 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:23.834 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:23.835 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:23.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.836 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:23.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.836 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:18:23.836 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:23.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 10 librados: watch_flush enter 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 10 librados: watch_flush exit 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 20 client.64530.objecter shutdown clearing up homeless session... 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 10 client.64530.objecter successfully canceled tick 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3a6403cea0 msgr2=0x7f3a6403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 --2- 172.21.15.138:0/376970292 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3a6403cea0 0x7f3a6403f360 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0078c0 tx=0x7f3a7c004f40 comp rx=0 tx=0).stop 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 10 monclient: shutdown 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409ef50 msgr2=0x7f3a84135a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 --2- 172.21.15.138:0/376970292 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3a8409ef50 0x7f3a84135a20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b2c0 tx=0x7f3a7401d330 comp rx=0 tx=0).stop 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a83fff640 1 -- 172.21.15.138:0/376970292 reap_dead start 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a83fff640 1 -- 172.21.15.138:0/376970292 reap_dead start 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 shutdown_connections 2024-09-11T05:18:23.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 >> 172.21.15.138:0/376970292 conn(0x7f3a84063640 msgr2=0x7f3a84072dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:23.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 shutdown_connections 2024-09-11T05:18:23.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 -- 172.21.15.138:0/376970292 wait complete. 2024-09-11T05:18:23.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.816+0000 7f3a8a1c9640 1 librados: shutdown 2024-09-11T05:18:23.839 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:18:23.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:18:23.839 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:18:23.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient: get_monmap_and_config 2024-09-11T05:18:23.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient: build_initial_monmap 2024-09-11T05:18:23.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient: monmap: 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:23.955565+0000 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:23.955565+0000 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:23.951 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 1 Processor -- start 2024-09-11T05:18:23.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 1 -- start start 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient: init 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb8009da20 0x7fbb800a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient(hunting): picked mon.noname-b con 0x7fbb8009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb800a1e60 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbb800a22a0 0x7fbb80136910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient(hunting): picked mon.noname-a con 0x7fbb800a22a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb80064590 con 0x7fbb800a1e60 2024-09-11T05:18:23.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fbb8009de00 con 0x7fbb800a22a0 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbb800a2c70 con 0x7fbb8009da20 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb86d7b640 10 monclient(hunting): authenticate will time out at 1587.391654s 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb85318640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbb800a22a0 0x7fbb80136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb84b17640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb8009da20 0x7fbb800a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb85318640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbb800a22a0 0x7fbb80136910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43588/0 (socket says 172.21.15.138:43588) 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb84b17640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb8009da20 0x7fbb800a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36302/0 (socket says 172.21.15.138:36302) 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb85318640 1 -- 172.21.15.138:0/1926148088 learned_addr learned my addr 172.21.15.138:0/1926148088 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb7ffff640 1 --1- 172.21.15.138:0/1926148088 >> v1:172.21.15.138:6789/0 conn(0x7fbb800a1e60 0x7fbb80136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38436/0 (socket says 172.21.15.138:38436) 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb85318640 10 monclient(hunting): get_auth_request con 0x7fbb800a22a0 auth_method 0 2024-09-11T05:18:23.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb85318640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb85318640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb85318640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb84b17640 10 monclient(hunting): get_auth_request con 0x7fbb8009da20 auth_method 0 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb84b17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb84b17640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.952+0000 7fbb84b17640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 10 monclient(hunting): handle_auth_done global_id 64539 payload 274 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 1 -- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb8009da20 msgr2=0x7fbb800a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 1 --2- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb8009da20 0x7fbb800a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 1 -- 172.21.15.138:0/1926148088 >> v1:172.21.15.138:6789/0 conn(0x7fbb800a1e60 legacy=0x7fbb80136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 10 monclient: _finish_hunting 0 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 1 monclient: found mon.noname-a 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb85318640 1 -- 172.21.15.138:0/1926148088 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb800a2f50 con 0x7fbb800a22a0 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.956+0000 7fbb84b17640 1 --2- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb8009da20 0x7fbb800a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 1 -- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a22a0 msgr2=0x7fbb80136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 1 --2- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a22a0 0x7fbb80136910 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009870 tx=0x7fbb7001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fbdc08d0662d616 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 1 -- 172.21.15.138:0/1926148088 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb700b2070 con 0x7fbb800a22a0 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:23.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: dump: 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:18:23.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 1 -- 172.21.15.138:0/1926148088 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb700a9070 con 0x7fbb800a22a0 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 1 -- 172.21.15.138:0/1926148088 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb700ad880 con 0x7fbb800a22a0 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7f7fe640 10 monclient: dump: 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.967 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.968 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.968 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.968 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.968 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.968 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.968 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 5 monclient: authenticate success, global_id 64539 2024-09-11T05:18:23.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:23.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 4 set_mon_vals no callback set 2024-09-11T05:18:23.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient: shutdown 2024-09-11T05:18:23.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:23.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a22a0 msgr2=0x7fbb80136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 --2- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a22a0 0x7fbb80136910 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009870 tx=0x7fbb7001c420 comp rx=0 tx=0).stop 2024-09-11T05:18:23.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- 172.21.15.138:0/1926148088 shutdown_connections 2024-09-11T05:18:23.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 --2- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a22a0 0x7fbb80136910 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 --2- 172.21.15.138:0/1926148088 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb8009da20 0x7fbb800a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- 172.21.15.138:0/1926148088 >> 172.21.15.138:0/1926148088 conn(0x7fbb80063640 msgr2=0x7fbb80063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:23.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- 172.21.15.138:0/1926148088 shutdown_connections 2024-09-11T05:18:23.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- 172.21.15.138:0/1926148088 wait complete. 2024-09-11T05:18:23.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient: build_initial_monmap 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient: monmap: 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:23.971 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 librados: starting msgr at 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 librados: starting objecter 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 Processor -- start 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- start start 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 librados: setting wanted keys 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 librados: calling monclient init 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient: init 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:18:23.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb8009da20 0x7fbb800709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient(hunting): picked mon.noname-b con 0x7fbb8009da20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a1e60 0x7fbb80070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient(hunting): picked mon.noname-a con 0x7fbb800a1e60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb800a22a0 0x7fbb801ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient(hunting): picked mon.noname-c con 0x7fbb800a22a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbb80140ee0 con 0x7fbb800a1e60 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbb80140d60 con 0x7fbb8009da20 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbb80141060 con 0x7fbb800a22a0 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb86d7b640 10 monclient(hunting): authenticate will time out at 1587.399510s 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a1e60 0x7fbb80070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb800a22a0 0x7fbb801ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a1e60 0x7fbb80070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43598/0 (socket says 172.21.15.138:43598) 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 -- 172.21.15.138:0/2351398561 learned_addr learned my addr 172.21.15.138:0/2351398561 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient(hunting): get_auth_request con 0x7fbb800a1e60 auth_method 0 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 10 monclient(hunting): get_auth_request con 0x7fbb800a22a0 auth_method 0 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb85318640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb84b17640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb8009da20 0x7fbb800709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb84b17640 10 monclient(hunting): get_auth_request con 0x7fbb8009da20 auth_method 0 2024-09-11T05:18:23.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb84b17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb84b17640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb84b17640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_done global_id 64542 payload 824 2024-09-11T05:18:23.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb800a22a0 msgr2=0x7fbb801ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb800a22a0 0x7fbb801ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb8009da20 msgr2=0x7fbb800709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb8009da20 0x7fbb800709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:23.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:18:23.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 monclient: found mon.noname-a 2024-09-11T05:18:23.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:23.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.960+0000 7fbb7ffff640 1 -- 172.21.15.138:0/2351398561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8009e110 con 0x7fbb800a1e60 2024-09-11T05:18:23.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7ffff640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a1e60 msgr2=0x7fbb80070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7ffff640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a1e60 0x7fbb80070ef0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800c430 tx=0x7fbb6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 client.?.objecter ms_handle_connect 0x7fbb800a1e60 2024-09-11T05:18:23.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:23.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68017070 con 0x7fbb800a1e60 2024-09-11T05:18:23.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: dump: 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: _finish_auth 0 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:18:23.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb68005820 con 0x7fbb800a1e60 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68005c10 con 0x7fbb800a1e60 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 monclient: dump: 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.984 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 5 monclient: authenticate success, global_id 64542 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7e7fc640 4 set_mon_vals no callback set 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 10 monclient: _renew_subs 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb80072270 con 0x7fbb800a1e60 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 10 client.64542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 10 monclient: _renew_subs 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb80134900 con 0x7fbb800a1e60 2024-09-11T05:18:23.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 1 librados: init done 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb80073ad0 0x7fbb801ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbb80073eb0 con 0x7fbb80073ad0 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb80073ad0 0x7fbb801ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 10 monclient: get_auth_request con 0x7fbb80073ad0 auth_method 0 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fbb6801b3b0 con 0x7fbb800a1e60 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 10 monclient(hunting): handle_auth_done global_id 84206 payload 824 2024-09-11T05:18:23.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 1 --2- 172.21.15.138:0/2351398561 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb6003cea0 0x7fbb6003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb84b17640 1 -- 172.21.15.138:0/2351398561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb80136b90 con 0x7fbb80073ad0 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7fbb6804f280 con 0x7fbb800a1e60 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 10 client.64542.objecter ms_dispatch 0x7fbb80000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 3 client.64542.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 3 client.64542.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.964+0000 7fbb7d7fa640 20 client.64542.objecter dump_active .. 0 homeless 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb85318640 1 --2- 172.21.15.138:0/2351398561 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb6003cea0 0x7fbb6003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb85318640 10 monclient: get_auth_request con 0x7fbb6003cea0 auth_method 0 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb84b17640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb80073ad0 msgr2=0x7fbb801ce950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb84b17640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb80073ad0 0x7fbb801ce950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007e90 tx=0x7fbb78011520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 10 client.64542.objecter ms_handle_connect 0x7fbb80073ad0 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 10 client.64542.objecter resend_mon_ops 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7801a020 con 0x7fbb80073ad0 2024-09-11T05:18:23.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 10 monclient: dump: 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbb78016800 con 0x7fbb80073ad0 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb80073ad0 msgr2=0x7fbb801ce950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb80073ad0 0x7fbb801ce950 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007e90 tx=0x7fbb78011520 comp rx=0 tx=0).stop 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb80073ad0 msgr2=0x7fbb801ce950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb80073ad0 0x7fbb801ce950 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007e90 tx=0x7fbb78011520 comp rx=0 tx=0).stop 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb7d7fa640 10 client.64542.objecter ms_handle_connect 0x7fbb6003cea0 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.968+0000 7fbb85318640 1 --2- 172.21.15.138:0/2351398561 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb6003cea0 0x7fbb6003f360 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7fbb700ac000 tx=0x7fbb700a7000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb86d7b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb86d7b640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb801cee90 0x7fbb8012daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb86d7b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbb800a1920 con 0x7fbb801cee90 2024-09-11T05:18:23.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb801cee90 0x7fbb8012daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 10 monclient: get_auth_request con 0x7fbb801cee90 auth_method 0 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 10 monclient(hunting): handle_auth_done global_id 84209 payload 824 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 1 -- 172.21.15.138:0/2351398561 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb8006cca0 con 0x7fbb801cee90 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb801cee90 msgr2=0x7fbb8012daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb84b17640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb801cee90 0x7fbb8012daa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007e90 tx=0x7fbb78004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 10 client.64542.objecter ms_handle_connect 0x7fbb801cee90 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 10 client.64542.objecter resend_mon_ops 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7801a030 con 0x7fbb801cee90 2024-09-11T05:18:23.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 10 monclient: dump: 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7fbb780039a0 con 0x7fbb801cee90 2024-09-11T05:18:23.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb801cee90 msgr2=0x7fbb8012daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb801cee90 0x7fbb8012daa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007e90 tx=0x7fbb78004a50 comp rx=0 tx=0).stop 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb801cee90 msgr2=0x7fbb8012daa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.976+0000 7fbb7d7fa640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbb801cee90 0x7fbb8012daa0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007e90 tx=0x7fbb78004a50 comp rx=0 tx=0).stop 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:18:23.992 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 40, 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 147746, 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:23.993 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:23.994 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:18:23.995 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:18:23.996 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:18:23.996 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:18:23.996 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:23.996 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:23.996 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:18:23.996 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:18:23.996 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:18:23.996 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:23.997 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:18:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:18:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:23.998 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:23.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:24.000 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:24.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:18:24.004 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.005 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.006 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:18:24.006 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:24.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 10 librados: watch_flush enter 2024-09-11T05:18:24.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 10 librados: watch_flush exit 2024-09-11T05:18:24.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 20 client.64542.objecter shutdown clearing up homeless session... 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 10 client.64542.objecter successfully canceled tick 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb6003cea0 msgr2=0x7fbb6003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 --2- 172.21.15.138:0/2351398561 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbb6003cea0 0x7fbb6003f360 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7fbb700ac000 tx=0x7fbb700a7000 comp rx=0 tx=0).stop 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 10 monclient: shutdown 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a1e60 msgr2=0x7fbb80070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb85318640 1 -- 172.21.15.138:0/2351398561 reap_dead start 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a1e60 0x7fbb80070ef0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbb6800c430 tx=0x7fbb6800c8f0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 shutdown_connections 2024-09-11T05:18:24.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 --2- 172.21.15.138:0/2351398561 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbb800a1e60 0x7fbb80070ef0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 >> 172.21.15.138:0/2351398561 conn(0x7fbb80063640 msgr2=0x7fbb800a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:24.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 shutdown_connections 2024-09-11T05:18:24.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 -- 172.21.15.138:0/2351398561 wait complete. 2024-09-11T05:18:24.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:23.980+0000 7fbb86d7b640 1 librados: shutdown 2024-09-11T05:18:24.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.010 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826098789}) 2024-09-11T05:18:24.010 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2111555651}) 2024-09-11T05:18:24.010 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.010 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826098789}) 2024-09-11T05:18:24.010 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:23.992+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587894907}) 2024-09-11T05:18:24.011 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:18:24.011 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:18:24.011 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:18:24.086 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.088+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.086 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.088+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=200134051,mdsmap=1754867906,mgr=772299015}) 2024-09-11T05:18:24.086 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.088+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=3407392990,mdsmap=1754867906,mgr=4007307362}) 2024-09-11T05:18:24.086 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.088+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.086 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.088+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=200134051,mdsmap=1754867906,mgr=772299015}) 2024-09-11T05:18:24.086 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.088+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=1559275048,mdsmap=1754867906,mgr=875472461}) 2024-09-11T05:18:24.100 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.100 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=32} crc {mgr=1747058506,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=109150555,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3654088858,osd_snap=3062036323,osdmap=2913036987}) 2024-09-11T05:18:24.100 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=36} crc {mgr=1747058506,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2321029162,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3654088858,osd_snap=3255672464,osdmap=3951772663}) 2024-09-11T05:18:24.100 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.101 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=32} crc {mgr=1747058506,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=109150555,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3654088858,osd_snap=3062036323,osdmap=2913036987}) 2024-09-11T05:18:24.101 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.100+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=33} crc {mgr=1747058506,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=109150555,mon_config_key=2831613776,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3654088858,osd_snap=2228480255,osdmap=3727159491}) 2024-09-11T05:18:24.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient: get_monmap_and_config 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient: build_initial_monmap 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient: monmap: 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:24.134708+0000 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:24.134708+0000 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:24.130 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 1 Processor -- start 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 1 -- start start 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient: init 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b40a32c0 0x7f88b409e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient(hunting): picked mon.noname-b con 0x7f88b40a32c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88b409e960 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f88b409ed80 0x7f88b4135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient(hunting): picked mon.noname-c con 0x7f88b409ed80 addr v2:172.21.15.190:3300/0 2024-09-11T05:18:24.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b409f230 con 0x7f88b409e960 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f88b40a1920 con 0x7f88b409ed80 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f88b40a36a0 con 0x7f88b40a32c0 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88baeda640 10 monclient(hunting): authenticate will time out at 1587.570888s 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b40a32c0 0x7f88b409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b3fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f88b409e960 0x7f88b41356f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38444/0 (socket says 172.21.15.138:38444) 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b3fff640 1 -- 172.21.15.138:0/1298730193 learned_addr learned my addr 172.21.15.138:0/1298730193 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient(hunting): get_auth_request con 0x7f88b40a32c0 auth_method 0 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient(hunting): handle_auth_done global_id 84212 payload 274 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 1 -- 172.21.15.138:0/1298730193 >> v2:172.21.15.190:3300/0 conn(0x7f88b409ed80 msgr2=0x7f88b4135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b9477640 1 --2- 172.21.15.138:0/1298730193 >> v2:172.21.15.190:3300/0 conn(0x7f88b409ed80 0x7f88b4135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 1 --2- 172.21.15.138:0/1298730193 >> v2:172.21.15.190:3300/0 conn(0x7f88b409ed80 0x7f88b4135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 1 -- 172.21.15.138:0/1298730193 >> v1:172.21.15.138:6789/0 conn(0x7f88b409e960 legacy=0x7f88b41356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient: _finish_hunting 0 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 1 monclient: found mon.noname-b 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b8c76640 1 -- 172.21.15.138:0/1298730193 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b40a1b50 con 0x7f88b40a32c0 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b9477640 10 monclient: get_auth_request con 0x7f88b409ed80 auth_method 0 2024-09-11T05:18:24.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.132+0000 7f88b9477640 1 --2- 172.21.15.138:0/1298730193 >> v2:172.21.15.190:3300/0 conn(0x7f88b409ed80 0x7f88b4135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:18:24.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b8c76640 1 -- 172.21.15.138:0/1298730193 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b40a32c0 msgr2=0x7f88b409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b8c76640 1 --2- 172.21.15.138:0/1298730193 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b40a32c0 0x7f88b409e420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d329c8f66beda3d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 1 -- 172.21.15.138:0/1298730193 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a40b0070 con 0x7f88b40a32c0 2024-09-11T05:18:24.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: dump: 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: _finish_auth 0 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: _check_auth_tickets 2024-09-11T05:18:24.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 1 -- 172.21.15.138:0/1298730193 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a40ab600 con 0x7f88b40a32c0 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88baeda640 5 monclient: authenticate success, global_id 84212 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88baeda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 1 -- 172.21.15.138:0/1298730193 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a40ab9d0 con 0x7f88b40a32c0 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88b37fe640 10 monclient: dump: 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.139 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88baeda640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.136+0000 7f88baeda640 4 set_mon_vals no callback set 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient: shutdown 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:24.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- 172.21.15.138:0/1298730193 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b40a32c0 msgr2=0x7f88b409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 --2- 172.21.15.138:0/1298730193 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b40a32c0 0x7f88b409e420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a401c330 comp rx=0 tx=0).stop 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- 172.21.15.138:0/1298730193 shutdown_connections 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 --2- 172.21.15.138:0/1298730193 >> v2:172.21.15.190:3300/0 conn(0x7f88b409ed80 0x7f88b4135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 --2- 172.21.15.138:0/1298730193 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b40a32c0 0x7f88b409e420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- 172.21.15.138:0/1298730193 >> 172.21.15.138:0/1298730193 conn(0x7f88b4063640 msgr2=0x7f88b4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- 172.21.15.138:0/1298730193 shutdown_connections 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- 172.21.15.138:0/1298730193 wait complete. 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient: build_initial_monmap 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient: monmap: 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:24.141 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 librados: starting msgr at 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 librados: starting objecter 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 Processor -- start 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- start start 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 librados: setting wanted keys 2024-09-11T05:18:24.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 librados: calling monclient init 2024-09-11T05:18:24.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient: init 2024-09-11T05:18:24.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:24.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:18:24.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b409e960 0x7f88b411da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient(hunting): picked mon.noname-c con 0x7f88b409e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:24.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88b409ed80 0x7f88b4135480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient(hunting): picked mon.noname-a con 0x7f88b409ed80 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:24.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b40a32c0 0x7f88b41ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient(hunting): picked mon.noname-b con 0x7f88b40a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f88b40a12d0 con 0x7f88b409ed80 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88b406e1f0 con 0x7f88b40a32c0 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f88b4120eb0 con 0x7f88b409e960 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88baeda640 10 monclient(hunting): authenticate will time out at 1587.577994s 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b409e960 0x7f88b411da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b409e960 0x7f88b411da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36370/0 (socket says 172.21.15.138:36370) 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88b409ed80 0x7f88b4135480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 1 -- 172.21.15.138:0/2369347698 learned_addr learned my addr 172.21.15.138:0/2369347698 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 10 monclient(hunting): get_auth_request con 0x7f88b409e960 auth_method 0 2024-09-11T05:18:24.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient(hunting): get_auth_request con 0x7f88b409ed80 auth_method 0 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b9477640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b40a32c0 0x7f88b41ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b9477640 10 monclient(hunting): get_auth_request con 0x7f88b40a32c0 auth_method 0 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b9477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b9477640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b9477640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient(hunting): handle_auth_done global_id 64545 payload 824 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b409e960 msgr2=0x7f88b411da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88b409e960 0x7f88b411da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b40a32c0 msgr2=0x7f88b41ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b40a32c0 0x7f88b41ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient: _finish_hunting 0 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 1 monclient: found mon.noname-a 2024-09-11T05:18:24.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.140+0000 7f88b3fff640 1 -- 172.21.15.138:0/2369347698 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b411ed20 con 0x7f88b409ed80 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b3fff640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88b409ed80 msgr2=0x7f88b4135480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b3fff640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88b409ed80 0x7f88b4135480 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f88a800b460 tx=0x7f88a800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 client.?.objecter ms_handle_connect 0x7f88b409ed80 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8015020 con 0x7f88b409ed80 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: dump: 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.153 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: _finish_auth 0 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: _check_auth_tickets 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a8011a60 con 0x7f88b409ed80 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:24.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8011e50 con 0x7f88b409ed80 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 monclient: dump: 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b27fc640 4 set_mon_vals no callback set 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 5 monclient: authenticate success, global_id 64545 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 10 monclient: _renew_subs 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b409e110 con 0x7f88b409ed80 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 10 client.64545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 10 monclient: _renew_subs 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b411df60 con 0x7f88b409ed80 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 1 librados: init done 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:24.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41359c0 0x7f88b41ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88b40a1450 con 0x7f88b41359c0 2024-09-11T05:18:24.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f88a801a400 con 0x7f88b409ed80 2024-09-11T05:18:24.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41359c0 0x7f88b41ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 10 monclient: get_auth_request con 0x7f88b41359c0 auth_method 0 2024-09-11T05:18:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 1 --2- 172.21.15.138:0/2369347698 >> v2:172.21.15.190:6800/1562782189 conn(0x7f88a003d040 0x7f88a003f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f88a804fc80 con 0x7f88b409ed80 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 10 client.64545.objecter ms_dispatch 0x7f88b4000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 3 client.64545.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 3 client.64545.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b17fa640 20 client.64545.objecter dump_active .. 0 homeless 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 10 monclient(hunting): handle_auth_done global_id 94132 payload 824 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b8c76640 1 -- 172.21.15.138:0/2369347698 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88b409e420 con 0x7f88b41359c0 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b9477640 1 --2- 172.21.15.138:0/2369347698 >> v2:172.21.15.190:6800/1562782189 conn(0x7f88a003d040 0x7f88a003f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.144+0000 7f88b9477640 10 monclient: get_auth_request con 0x7f88a003d040 auth_method 0 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b8c76640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41359c0 msgr2=0x7f88b41ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b8c76640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41359c0 0x7f88b41ce5c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a40190d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 10 client.64545.objecter ms_handle_connect 0x7f88b41359c0 2024-09-11T05:18:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 10 client.64545.objecter resend_mon_ops 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a40bb070 con 0x7f88b41359c0 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 10 monclient: dump: 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 10 client.64545.objecter ms_handle_connect 0x7f88a003d040 2024-09-11T05:18:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b9477640 1 --2- 172.21.15.138:0/2369347698 >> v2:172.21.15.190:6800/1562782189 conn(0x7f88a003d040 0x7f88a003f500 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7f88b4071ee0 tx=0x7f88ac007540 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f88a40b5610 con 0x7f88b41359c0 2024-09-11T05:18:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41359c0 msgr2=0x7f88b41ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41359c0 0x7f88b41ce5c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a40190d0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41359c0 msgr2=0x7f88b41ce5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.148+0000 7f88b17fa640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41359c0 0x7f88b41ce5c0 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a40190d0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88baeda640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88baeda640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41ceb00 0x7f88b4138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88baeda640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88b41cef10 con 0x7f88b41ceb00 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41ceb00 0x7f88b4138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 10 monclient: get_auth_request con 0x7f88b41ceb00 auth_method 0 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 10 monclient(hunting): handle_auth_done global_id 94135 payload 824 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.156+0000 7f88b8c76640 1 -- 172.21.15.138:0/2369347698 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88b41205e0 con 0x7f88b41ceb00 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b8c76640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41ceb00 msgr2=0x7f88b4138920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b8c76640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41ceb00 0x7f88b4138920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f88b4074da0 tx=0x7f88a40a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 10 client.64545.objecter ms_handle_connect 0x7f88b41ceb00 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 10 client.64545.objecter resend_mon_ops 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a40b0040 con 0x7f88b41ceb00 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:18:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 10 monclient: dump: 2024-09-11T05:18:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f88a40b5e10 con 0x7f88b41ceb00 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41ceb00 msgr2=0x7f88b4138920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41ceb00 0x7f88b4138920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f88b4074da0 tx=0x7f88a40a8000 comp rx=0 tx=0).stop 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41ceb00 msgr2=0x7f88b4138920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b17fa640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88b41ceb00 0x7f88b4138920 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f88b4074da0 tx=0x7f88a40a8000 comp rx=0 tx=0).stop 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 40, 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 18107, 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:24.170 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 3, 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:24.171 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:24.172 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:18:24.173 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:24.174 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:24.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:24.177 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:24.179 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.180 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:24.181 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:24.182 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:18:24.183 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:24.183 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.183 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3035395478}) 2024-09-11T05:18:24.183 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3841232706}) 2024-09-11T05:18:24.183 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.184 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3035395478}) 2024-09-11T05:18:24.184 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1634121941}) 2024-09-11T05:18:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 10 librados: watch_flush enter 2024-09-11T05:18:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 10 librados: watch_flush exit 2024-09-11T05:18:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 20 client.64545.objecter shutdown clearing up homeless session... 2024-09-11T05:18:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 10 client.64545.objecter successfully canceled tick 2024-09-11T05:18:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 >> v2:172.21.15.190:6800/1562782189 conn(0x7f88a003d040 msgr2=0x7f88a003f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 --2- 172.21.15.138:0/2369347698 >> v2:172.21.15.190:6800/1562782189 conn(0x7f88a003d040 0x7f88a003f500 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7f88b4071ee0 tx=0x7f88ac007540 comp rx=0 tx=0).stop 2024-09-11T05:18:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 10 monclient: shutdown 2024-09-11T05:18:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88b409ed80 msgr2=0x7f88b4135480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 --2- 172.21.15.138:0/2369347698 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88b409ed80 0x7f88b4135480 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f88a800b460 tx=0x7f88a800b920 comp rx=0 tx=0).stop 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b9477640 1 -- 172.21.15.138:0/2369347698 reap_dead start 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88b9477640 1 -- 172.21.15.138:0/2369347698 reap_dead start 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 shutdown_connections 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 >> 172.21.15.138:0/2369347698 conn(0x7f88b4063640 msgr2=0x7f88b40a67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 shutdown_connections 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 -- 172.21.15.138:0/2369347698 wait complete. 2024-09-11T05:18:24.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.160+0000 7f88baeda640 1 librados: shutdown 2024-09-11T05:18:24.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:18:24.188 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T05:18:24.188 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T05:18:24.188 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T05:18:24.188 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-11T05:18:24.188 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient: get_monmap_and_config 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient: build_initial_monmap 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient: monmap: 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:24.311203+0000 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:24.311203+0000 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:24.306 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.307 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:24.307 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:24.307 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:24.307 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:24.307 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:24.307 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 1 Processor -- start 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 1 -- start start 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient: init 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f809da20 0x7f80f80a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient(hunting): picked mon.noname-b con 0x7f80f809da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80f80a1f40 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f80f80a23a0 0x7f80f8136880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient(hunting): picked mon.noname-c con 0x7f80f80a23a0 addr v2:172.21.15.190:3300/0 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f8074a80 con 0x7f80f80a1f40 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f80f80a2830 con 0x7f80f80a23a0 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f80f80a2d90 con 0x7f80f809da20 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80ff5a5640 10 monclient(hunting): authenticate will time out at 1587.747459s 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fcb40640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f80f80a1f40 0x7f80f8136170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38448/0 (socket says 172.21.15.138:38448) 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fcb40640 1 -- 172.21.15.138:0/1971527571 learned_addr learned my addr 172.21.15.138:0/1971527571 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f809da20 0x7f80f80a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient(hunting): get_auth_request con 0x7f80f809da20 auth_method 0 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fdb42640 1 --2- 172.21.15.138:0/1971527571 >> v2:172.21.15.190:3300/0 conn(0x7f80f80a23a0 0x7f80f8136880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fdb42640 10 monclient(hunting): get_auth_request con 0x7f80f80a23a0 auth_method 0 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fdb42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fdb42640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fdb42640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient(hunting): handle_auth_done global_id 84218 payload 274 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 1 -- 172.21.15.138:0/1971527571 >> v2:172.21.15.190:3300/0 conn(0x7f80f80a23a0 msgr2=0x7f80f8136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 1 --2- 172.21.15.138:0/1971527571 >> v2:172.21.15.190:3300/0 conn(0x7f80f80a23a0 0x7f80f8136880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 1 -- 172.21.15.138:0/1971527571 >> v1:172.21.15.138:6789/0 conn(0x7f80f80a1f40 legacy=0x7f80f8136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.308+0000 7f80fd341640 10 monclient: _finish_hunting 0 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.312+0000 7f80fd341640 1 monclient: found mon.noname-b 2024-09-11T05:18:24.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.312+0000 7f80fd341640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:24.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.312+0000 7f80fd341640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.312+0000 7f80fd341640 1 -- 172.21.15.138:0/1971527571 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f80a3070 con 0x7f80f809da20 2024-09-11T05:18:24.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fd341640 1 -- 172.21.15.138:0/1971527571 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f809da20 msgr2=0x7f80f80a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fd341640 1 --2- 172.21.15.138:0/1971527571 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f809da20 0x7f80f80a1a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f80f00097c0 tx=0x7f80f001c150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a655d54e16de server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 1 -- 172.21.15.138:0/1971527571 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f00b0070 con 0x7f80f809da20 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: dump: 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.313 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: _finish_auth 0 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: _check_auth_tickets 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 5 monclient: authenticate success, global_id 84218 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 1 -- 172.21.15.138:0/1971527571 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f00ab600 con 0x7f80f809da20 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 4 set_mon_vals no callback set 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 1 -- 172.21.15.138:0/1971527571 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f00ab9d0 con 0x7f80f809da20 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:24.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80f7fff640 10 monclient: dump: 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.315 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient: shutdown 2024-09-11T05:18:24.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:24.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- 172.21.15.138:0/1971527571 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f809da20 msgr2=0x7f80f80a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 --2- 172.21.15.138:0/1971527571 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f809da20 0x7f80f80a1a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f80f00097c0 tx=0x7f80f001c150 comp rx=0 tx=0).stop 2024-09-11T05:18:24.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- 172.21.15.138:0/1971527571 shutdown_connections 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 --2- 172.21.15.138:0/1971527571 >> v2:172.21.15.190:3300/0 conn(0x7f80f80a23a0 0x7f80f8136880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 --2- 172.21.15.138:0/1971527571 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f809da20 0x7f80f80a1a00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- 172.21.15.138:0/1971527571 >> 172.21.15.138:0/1971527571 conn(0x7f80f80a3b00 msgr2=0x7f80f80a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- 172.21.15.138:0/1971527571 shutdown_connections 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- 172.21.15.138:0/1971527571 wait complete. 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient: build_initial_monmap 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient: monmap: 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:24.323 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 librados: starting msgr at 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 librados: starting objecter 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 Processor -- start 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- start start 2024-09-11T05:18:24.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 librados: setting wanted keys 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 librados: calling monclient init 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient: init 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f809da20 0x7f80f8073b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient(hunting): picked mon.noname-b con 0x7f80f809da20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f80a1f40 0x7f80f809e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient(hunting): picked mon.noname-c con 0x7f80f80a1f40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80f80a23a0 0x7f80f809ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient(hunting): picked mon.noname-a con 0x7f80f80a23a0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f80f8140eb0 con 0x7f80f80a23a0 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f80f8140d30 con 0x7f80f809da20 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f80f8141030 con 0x7f80f80a1f40 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80ff5a5640 10 monclient(hunting): authenticate will time out at 1587.754484s 2024-09-11T05:18:24.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f80a1f40 0x7f80f809e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f80a1f40 0x7f80f809e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36386/0 (socket says 172.21.15.138:36386) 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 -- 172.21.15.138:0/2031075060 learned_addr learned my addr 172.21.15.138:0/2031075060 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fdb42640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80f80a23a0 0x7f80f809ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient(hunting): get_auth_request con 0x7f80f80a1f40 auth_method 0 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fdb42640 10 monclient(hunting): get_auth_request con 0x7f80f80a23a0 auth_method 0 2024-09-11T05:18:24.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fdb42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fdb42640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fdb42640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fdb42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fdb42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fd341640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f809da20 0x7f80f8073b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fdb42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fd341640 10 monclient(hunting): get_auth_request con 0x7f80f809da20 auth_method 0 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fd341640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fd341640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fd341640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient(hunting): handle_auth_done global_id 84221 payload 824 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f809da20 msgr2=0x7f80f8073b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f809da20 0x7f80f8073b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80f80a23a0 msgr2=0x7f80f809ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f80f80a23a0 0x7f80f809ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient: _finish_hunting 0 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 monclient: found mon.noname-c 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.316+0000 7f80fcb40640 1 -- 172.21.15.138:0/2031075060 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f813cf40 con 0x7f80f80a1f40 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fcb40640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f80a1f40 msgr2=0x7f80f809e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fcb40640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f80a1f40 0x7f80f809e530 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00cc10 tx=0x7f80ec00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 client.?.objecter ms_handle_connect 0x7f80f80a1f40 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec012400 con 0x7f80f80a1f40 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: dump: 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: _finish_auth 0 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80ec0125e0 con 0x7f80f80a1f40 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec0129d0 con 0x7f80f80a1f40 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 5 monclient: authenticate success, global_id 84221 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 10 monclient: _renew_subs 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f813d220 con 0x7f80f80a1f40 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 monclient: dump: 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.331 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f6ffd640 4 set_mon_vals no callback set 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 10 client.84221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 10 monclient: _renew_subs 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f809e1f0 con 0x7f80f80a1f40 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 1 librados: init done 2024-09-11T05:18:24.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f802e5d0 0x7f80f809efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f80f802e9b0 con 0x7f80f802e5d0 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f80ec012bf0 con 0x7f80f80a1f40 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f802e5d0 0x7f80f809efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 10 monclient: get_auth_request con 0x7f80f802e5d0 auth_method 0 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 1 --2- 172.21.15.138:0/2031075060 >> v2:172.21.15.190:6800/1562782189 conn(0x7f80e403cea0 0x7f80e403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f80ec0501b0 con 0x7f80f80a1f40 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 10 client.84221.objecter ms_dispatch 0x7f80f8000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 3 client.84221.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 3 client.84221.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80f5ffb640 20 client.84221.objecter dump_active .. 0 homeless 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fdb42640 1 --2- 172.21.15.138:0/2031075060 >> v2:172.21.15.190:6800/1562782189 conn(0x7f80e403cea0 0x7f80e403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fdb42640 10 monclient: get_auth_request con 0x7f80e403cea0 auth_method 0 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 10 monclient(hunting): handle_auth_done global_id 94144 payload 824 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.320+0000 7f80fd341640 1 -- 172.21.15.138:0/2031075060 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80f80a1a00 con 0x7f80f802e5d0 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 10 client.84221.objecter ms_handle_connect 0x7f80f802e5d0 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80fd341640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f802e5d0 msgr2=0x7f80f809efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 10 client.84221.objecter resend_mon_ops 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80fd341640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f802e5d0 0x7f80f809efb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f80f00ad000 tx=0x7f80f0018ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f00bb070 con 0x7f80f802e5d0 2024-09-11T05:18:24.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80fdb42640 1 --2- 172.21.15.138:0/2031075060 >> v2:172.21.15.190:6800/1562782189 conn(0x7f80e403cea0 0x7f80e403f360 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7f80f81363d0 tx=0x7f80e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 10 monclient: dump: 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 10 client.84221.objecter ms_handle_connect 0x7f80e403cea0 2024-09-11T05:18:24.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f80f00c0450 con 0x7f80f802e5d0 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f802e5d0 msgr2=0x7f80f809efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f802e5d0 0x7f80f809efb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f80f00ad000 tx=0x7f80f0018ef0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f802e5d0 msgr2=0x7f80f809efb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.324+0000 7f80f5ffb640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f802e5d0 0x7f80f809efb0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f80f00ad000 tx=0x7f80f0018ef0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80ff5a5640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80ff5a5640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f8063e00 0x7f80f80641e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80ff5a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f80f812f070 con 0x7f80f8063e00 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f8063e00 0x7f80f80641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 10 monclient: get_auth_request con 0x7f80f8063e00 auth_method 0 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 10 monclient(hunting): handle_auth_done global_id 94147 payload 824 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.332+0000 7f80fd341640 1 -- 172.21.15.138:0/2031075060 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f80f813e1b0 con 0x7f80f8063e00 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 10 client.84221.objecter ms_handle_connect 0x7f80f8063e00 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 10 client.84221.objecter resend_mon_ops 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80fd341640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f8063e00 msgr2=0x7f80f80641e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80fd341640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f8063e00 0x7f80f80641e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f80f001bf80 tx=0x7f80f0009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f00ce080 con 0x7f80f8063e00 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 10 monclient: dump: 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f80f0003e10 con 0x7f80f8063e00 2024-09-11T05:18:24.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f8063e00 msgr2=0x7f80f80641e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f8063e00 0x7f80f80641e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f80f001bf80 tx=0x7f80f0009190 comp rx=0 tx=0).stop 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f8063e00 msgr2=0x7f80f80641e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80f5ffb640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f80f8063e00 0x7f80f80641e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "ret": 0, 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T05:18:24.341 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:24.342 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.342 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2928215674}) 2024-09-11T05:18:24.342 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1855289677}) 2024-09-11T05:18:24.342 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.342 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2928215674}) 2024-09-11T05:18:24.342 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2732712714}) 2024-09-11T05:18:24.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 10 librados: watch_flush enter 2024-09-11T05:18:24.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 10 librados: watch_flush exit 2024-09-11T05:18:24.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 20 client.84221.objecter shutdown clearing up homeless session... 2024-09-11T05:18:24.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 10 client.84221.objecter successfully canceled tick 2024-09-11T05:18:24.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 >> v2:172.21.15.190:6800/1562782189 conn(0x7f80e403cea0 msgr2=0x7f80e403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 --2- 172.21.15.138:0/2031075060 >> v2:172.21.15.190:6800/1562782189 conn(0x7f80e403cea0 0x7f80e403f360 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7f80f81363d0 tx=0x7f80e8008040 comp rx=0 tx=0).stop 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 10 monclient: shutdown 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f80a1f40 msgr2=0x7f80f809e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f80a1f40 0x7f80f809e530 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00cc10 tx=0x7f80ec00a430 comp rx=0 tx=0).stop 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80fdb42640 1 -- 172.21.15.138:0/2031075060 reap_dead start 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 shutdown_connections 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 --2- 172.21.15.138:0/2031075060 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f80f80a1f40 0x7f80f809e530 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 >> 172.21.15.138:0/2031075060 conn(0x7f80f80a3b00 msgr2=0x7f80f8132e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 shutdown_connections 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 -- 172.21.15.138:0/2031075060 wait complete. 2024-09-11T05:18:24.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.336+0000 7f80ff5a5640 1 librados: shutdown 2024-09-11T05:18:24.348 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.348+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.348 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.348+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=2065258215}) 2024-09-11T05:18:24.348 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.348+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=20} crc {osdmap=3423826665}) 2024-09-11T05:18:24.348 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.348+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:18:24.348 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.348+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=2065258215}) 2024-09-11T05:18:24.348 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:18:24.348+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=20} crc {osdmap=3423826665}) 2024-09-11T05:18:24.352 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T05:18:24.352 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T05:18:24.353 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:18:24.373 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T05:18:24.373 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:18:24.373 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:18:24.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient: get_monmap_and_config 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient: build_initial_monmap 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient: monmap: 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:24.494681+0000 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:24.494681+0000 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:24.490 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:24.491 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:24.491 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:24.491 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:24.491 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 1 Processor -- start 2024-09-11T05:18:24.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 1 -- start start 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient: init 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ed40a5b40 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a7130 0x7f0ed413db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed40a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0ed409d840 0x7f0ed4140090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed409d840 addr v2:172.21.15.190:3300/0 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed409d490 con 0x7f0ed40a5b40 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0ed409e110 con 0x7f0ed409d840 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed4070580 con 0x7f0ed40a7130 2024-09-11T05:18:24.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed89af640 10 monclient(hunting): authenticate will time out at 1587.930651s 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a7130 0x7f0ed413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a7130 0x7f0ed413db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36390/0 (socket says 172.21.15.138:36390) 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/1548534808 learned_addr learned my addr 172.21.15.138:0/1548534808 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient(hunting): get_auth_request con 0x7f0ed40a7130 auth_method 0 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed2d9d640 1 -- 172.21.15.138:0/1548534808 >> v2:172.21.15.190:3300/0 conn(0x7f0ed409d840 msgr2=0x7f0ed4140090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed259c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f0ed40a5b40 0x7f0ed40a5f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:38452/0 (socket says 172.21.15.138:38452) 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed2d9d640 1 --2- 172.21.15.138:0/1548534808 >> v2:172.21.15.190:3300/0 conn(0x7f0ed409d840 0x7f0ed4140090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_done global_id 84224 payload 274 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/1548534808 >> v2:172.21.15.190:3300/0 conn(0x7f0ed409d840 msgr2=0x7f0ed4140090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 1 --2- 172.21.15.138:0/1548534808 >> v2:172.21.15.190:3300/0 conn(0x7f0ed409d840 0x7f0ed4140090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/1548534808 >> v1:172.21.15.138:6789/0 conn(0x7f0ed40a5b40 legacy=0x7f0ed40a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 1 monclient: found mon.noname-b 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.492+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/1548534808 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed409cb80 con 0x7f0ed40a7130 2024-09-11T05:18:24.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/1548534808 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a7130 msgr2=0x7f0ed413db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed1d9b640 1 --2- 172.21.15.138:0/1548534808 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a7130 0x7f0ed413db40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009870 tx=0x7f0ebc01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9d6d3e286b6048 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 1 -- 172.21.15.138:0/1548534808 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0b0070 con 0x7f0ed40a7130 2024-09-11T05:18:24.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: dump: 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: _finish_auth 0 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: _check_auth_tickets 2024-09-11T05:18:24.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 1 -- 172.21.15.138:0/1548534808 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ebc0a4e50 con 0x7f0ed40a7130 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 1 -- 172.21.15.138:0/1548534808 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0ab820 con 0x7f0ed40a7130 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed89af640 5 monclient: authenticate success, global_id 84224 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed89af640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed89af640 4 set_mon_vals no callback set 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.496+0000 7f0ed159a640 10 monclient: dump: 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.503 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient: shutdown 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- 172.21.15.138:0/1548534808 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a7130 msgr2=0x7f0ed413db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 --2- 172.21.15.138:0/1548534808 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a7130 0x7f0ed413db40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009870 tx=0x7f0ebc01c440 comp rx=0 tx=0).stop 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- 172.21.15.138:0/1548534808 shutdown_connections 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 --2- 172.21.15.138:0/1548534808 >> v2:172.21.15.190:3300/0 conn(0x7f0ed409d840 0x7f0ed4140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 --2- 172.21.15.138:0/1548534808 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a7130 0x7f0ed413db40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- 172.21.15.138:0/1548534808 >> 172.21.15.138:0/1548534808 conn(0x7f0ed40aba10 msgr2=0x7f0ed40a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- 172.21.15.138:0/1548534808 shutdown_connections 2024-09-11T05:18:24.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- 172.21.15.138:0/1548534808 wait complete. 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient: build_initial_monmap 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient: monmap: 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 librados: starting msgr at 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 librados: starting objecter 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 Processor -- start 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- start start 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 librados: setting wanted keys 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 librados: calling monclient init 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient: init 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ed409d840 0x7f0ed41d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed409d840 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a5b40 0x7f0ed4138d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed40a5b40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed40a7130 0x7f0ed413de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed40a7130 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4143180 con 0x7f0ed409d840 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4143000 con 0x7f0ed40a7130 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed4143300 con 0x7f0ed40a5b40 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed89af640 10 monclient(hunting): authenticate will time out at 1587.937889s 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a5b40 0x7f0ed4138d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ed409d840 0x7f0ed41d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a5b40 0x7f0ed4138d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:36406/0 (socket says 172.21.15.138:36406) 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 10 monclient(hunting): get_auth_request con 0x7f0ed409d840 auth_method 0 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/4237008207 learned_addr learned my addr 172.21.15.138:0/4237008207 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient(hunting): get_auth_request con 0x7f0ed40a5b40 auth_method 0 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed2d9d640 1 -- 172.21.15.138:0/4237008207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed40a7130 msgr2=0x7f0ed413de30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed2d9d640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed40a7130 0x7f0ed413de30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:24.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient(hunting): handle_auth_done global_id 84227 payload 824 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/4237008207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed40a7130 msgr2=0x7f0ed413de30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed40a7130 0x7f0ed413de30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ed409d840 msgr2=0x7f0ed41d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ed409d840 0x7f0ed41d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 monclient: found mon.noname-c 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/4237008207 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed40a5460 con 0x7f0ed40a5b40 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.500+0000 7f0ed259c640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ed409d840 0x7f0ed41d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed1d9b640 1 -- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a5b40 msgr2=0x7f0ed4138d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed1d9b640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a5b40 0x7f0ed4138d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009870 tx=0x7f0ebc01ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 client.?.objecter ms_handle_connect 0x7f0ed40a5b40 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 1 -- 172.21.15.138:0/4237008207 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0ba020 con 0x7f0ed40a5b40 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: dump: 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.510 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: _finish_auth 0 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: _check_auth_tickets 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 1 -- 172.21.15.138:0/4237008207 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ebc0b6620 con 0x7f0ed40a5b40 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 5 monclient: authenticate success, global_id 84227 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 10 monclient: _renew_subs 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed4123700 con 0x7f0ed40a5b40 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 1 -- 172.21.15.138:0/4237008207 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0b0040 con 0x7f0ed40a5b40 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 monclient: dump: 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:24.512 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:24.513 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:24.513 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:24.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 10 client.84227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:24.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 10 monclient: _renew_subs 2024-09-11T05:18:24.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed4123420 con 0x7f0ed40a5b40 2024-09-11T05:18:24.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed0d99640 4 set_mon_vals no callback set 2024-09-11T05:18:24.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 1 librados: init done 2024-09-11T05:18:24.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:24.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:24.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed40a5f20 con 0x7f0ed40a5b40 2024-09-11T05:18:24.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 1 -- 172.21.15.138:0/4237008207 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f0ebc0b3030 con 0x7f0ed40a5b40 2024-09-11T05:18:24.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 1 --2- 172.21.15.138:0/4237008207 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0eb403cea0 0x7f0eb403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:24.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 1 -- 172.21.15.138:0/4237008207 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f0ebc0e9d60 con 0x7f0ed40a5b40 2024-09-11T05:18:24.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 10 client.84227.objecter ms_dispatch 0x7f0ed4000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 3 client.84227.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 3 client.84227.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ecaffd640 20 client.84227.objecter dump_active .. 0 homeless 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed259c640 1 --2- 172.21.15.138:0/4237008207 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0eb403cea0 0x7f0eb403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.504+0000 7f0ed259c640 10 monclient: get_auth_request con 0x7f0eb403cea0 auth_method 0 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.508+0000 7f0ecaffd640 10 client.84227.objecter ms_handle_connect 0x7f0eb403cea0 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.508+0000 7f0ed259c640 1 --2- 172.21.15.138:0/4237008207 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0eb403cea0 0x7f0eb403f360 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f0ed409e5b0 tx=0x7f0ecc004fc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.508+0000 7f0ecaffd640 1 -- 172.21.15.138:0/4237008207 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f0ebc0ef020 con 0x7f0ed40a5b40 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.508+0000 7f0ecaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:24.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.508+0000 7f0ecaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:24.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:18:24.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:24.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:24.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0ed40a3fd0 con 0x7f0ed40a5b40 2024-09-11T05:18:24.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ecaffd640 1 -- 172.21.15.138:0/4237008207 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f0ebc0c4030 con 0x7f0ed40a5b40 2024-09-11T05:18:24.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ecaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:24.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ecaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:24.741 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:18:24.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 10 librados: watch_flush enter 2024-09-11T05:18:24.743 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 10 librados: watch_flush exit 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 20 client.84227.objecter shutdown clearing up homeless session... 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 10 client.84227.objecter successfully canceled tick 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0eb403cea0 msgr2=0x7f0eb403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 --2- 172.21.15.138:0/4237008207 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0eb403cea0 0x7f0eb403f360 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f0ed409e5b0 tx=0x7f0ecc004fc0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 10 monclient: shutdown 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a5b40 msgr2=0x7f0ed4138d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a5b40 0x7f0ed4138d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009870 tx=0x7f0ebc01ac80 comp rx=0 tx=0).stop 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 shutdown_connections 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 --2- 172.21.15.138:0/4237008207 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0eb403cea0 0x7f0eb403f360 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ed40a7130 0x7f0ed413de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.744 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ed40a5b40 0x7f0ed4138d70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 --2- 172.21.15.138:0/4237008207 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ed409d840 0x7f0ed41d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:24.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 >> 172.21.15.138:0/4237008207 conn(0x7f0ed40aba10 msgr2=0x7f0ed413fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:24.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 shutdown_connections 2024-09-11T05:18:24.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 -- 172.21.15.138:0/4237008207 wait complete. 2024-09-11T05:18:24.745 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:24.744+0000 7f0ed89af640 1 librados: shutdown 2024-09-11T05:18:27.760 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:18:27.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient: get_monmap_and_config 2024-09-11T05:18:27.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient: build_initial_monmap 2024-09-11T05:18:27.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient: monmap: 2024-09-11T05:18:27.886 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:27.886 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:27.891343+0000 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:27.891343+0000 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:27.887 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:27.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 1 Processor -- start 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 1 -- start start 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient: init 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fab400a0a90 0x7fab400a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient(hunting): picked mon.noname-a con 0x7fab400a0a90 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab4009dc20 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4013bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient(hunting): picked mon.noname-b con 0x7fab40139be0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4009efa0 con 0x7fab4009dc20 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fab400a37b0 con 0x7fab400a0a90 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:27.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fab400a2d80 con 0x7fab40139be0 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab47a1f640 10 monclient(hunting): authenticate will time out at 1591.327577s 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fab400a0a90 0x7fab400a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab45fbc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4013bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab44fba640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fab4009dc20 0x7fab400a2670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:35794/0 (socket says 172.21.15.138:35794) 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fab400a0a90 0x7fab400a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:41830/0 (socket says 172.21.15.138:41830) 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab44fba640 1 -- 172.21.15.138:0/3833352126 learned_addr learned my addr 172.21.15.138:0/3833352126 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab45fbc640 10 monclient(hunting): get_auth_request con 0x7fab40139be0 auth_method 0 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab45fbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab45fbc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab45fbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 10 monclient(hunting): get_auth_request con 0x7fab400a0a90 auth_method 0 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:27.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab45fbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab45fbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab45fbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.888+0000 7fab457bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 10 monclient(hunting): handle_auth_done global_id 84230 payload 274 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 1 -- 172.21.15.138:0/3833352126 >> v2:172.21.15.138:3300/0 conn(0x7fab400a0a90 msgr2=0x7fab400a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 1 --2- 172.21.15.138:0/3833352126 >> v2:172.21.15.138:3300/0 conn(0x7fab400a0a90 0x7fab400a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 1 -- 172.21.15.138:0/3833352126 >> v1:172.21.15.138:6789/0 conn(0x7fab4009dc20 legacy=0x7fab400a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 10 monclient: _finish_hunting 0 2024-09-11T05:18:27.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 1 monclient: found mon.noname-b 2024-09-11T05:18:27.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:27.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:18:27.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab45fbc640 1 -- 172.21.15.138:0/3833352126 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4009ecc0 con 0x7fab40139be0 2024-09-11T05:18:27.893 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.892+0000 7fab457bb640 1 --2- 172.21.15.138:0/3833352126 >> v2:172.21.15.138:3300/0 conn(0x7fab400a0a90 0x7fab400a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:27.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 -- 172.21.15.138:0/3833352126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 msgr2=0x7fab4013bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:27.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 --2- 172.21.15.138:0/3833352126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4013bfd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab38009d90 tx=0x7fab3801c6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f3a6b172bb0bf73 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:27.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 1 -- 172.21.15.138:0/3833352126 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380ac070 con 0x7fab40139be0 2024-09-11T05:18:27.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: dump: 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:27.898 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: _finish_auth 0 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: _check_auth_tickets 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 1 -- 172.21.15.138:0/3833352126 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab380a66a0 con 0x7fab40139be0 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 5 monclient: authenticate success, global_id 84230 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 1 -- 172.21.15.138:0/3833352126 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380a6a70 con 0x7fab40139be0 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:27.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 4 set_mon_vals no callback set 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab37fff640 10 monclient: dump: 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:27.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient: shutdown 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- 172.21.15.138:0/3833352126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 msgr2=0x7fab4013bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3833352126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4013bfd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab38009d90 tx=0x7fab3801c6b0 comp rx=0 tx=0).stop 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- 172.21.15.138:0/3833352126 shutdown_connections 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3833352126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4013bfd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3833352126 >> v2:172.21.15.138:3300/0 conn(0x7fab400a0a90 0x7fab400a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- 172.21.15.138:0/3833352126 >> 172.21.15.138:0/3833352126 conn(0x7fab40063640 msgr2=0x7fab40063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- 172.21.15.138:0/3833352126 shutdown_connections 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- 172.21.15.138:0/3833352126 wait complete. 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient: build_initial_monmap 2024-09-11T05:18:27.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient: monmap: 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 librados: starting msgr at 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 librados: starting objecter 2024-09-11T05:18:27.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 Processor -- start 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- start start 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 librados: setting wanted keys 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 librados: calling monclient init 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient: init 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab4009dc20 0x7fab4011e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient(hunting): picked mon.noname-b con 0x7fab4009dc20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab400a0a90 0x7fab4011e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient(hunting): picked mon.noname-a con 0x7fab400a0a90 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4011ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient(hunting): picked mon.noname-c con 0x7fab40139be0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:27.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fab4013ca40 con 0x7fab400a0a90 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fab4013c8c0 con 0x7fab4009dc20 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fab4013cbc0 con 0x7fab40139be0 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab47a1f640 10 monclient(hunting): authenticate will time out at 1591.334682s 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab400a0a90 0x7fab4011e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4011ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4011ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:51052/0 (socket says 172.21.15.138:51052) 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab400a0a90 0x7fab4011e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:41832/0 (socket says 172.21.15.138:41832) 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 -- 172.21.15.138:0/3692842249 learned_addr learned my addr 172.21.15.138:0/3692842249 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient(hunting): get_auth_request con 0x7fab40139be0 auth_method 0 2024-09-11T05:18:27.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 10 monclient(hunting): get_auth_request con 0x7fab400a0a90 auth_method 0 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab457bb640 1 -- 172.21.15.138:0/3692842249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab4009dc20 msgr2=0x7fab4011e1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab457bb640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab4009dc20 0x7fab4011e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:27.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient(hunting): handle_auth_done global_id 84233 payload 824 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 -- 172.21.15.138:0/3692842249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab4009dc20 msgr2=0x7fab4011e1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab4009dc20 0x7fab4011e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 -- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab400a0a90 msgr2=0x7fab4011e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab400a0a90 0x7fab4011e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient: _finish_hunting 0 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 monclient: found mon.noname-c 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab45fbc640 1 -- 172.21.15.138:0/3692842249 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab40121f20 con 0x7fab40139be0 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.896+0000 7fab44fba640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab400a0a90 0x7fab4011e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab45fbc640 1 -- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 msgr2=0x7fab4011ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab45fbc640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4011ec20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fab3801cf50 tx=0x7fab38019450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:27.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 client.?.objecter ms_handle_connect 0x7fab40139be0 2024-09-11T05:18:27.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:27.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 1 -- 172.21.15.138:0/3692842249 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380c0070 con 0x7fab40139be0 2024-09-11T05:18:27.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:27.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:18:27.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: dump: 2024-09-11T05:18:27.909 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: _finish_auth 0 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 1 -- 172.21.15.138:0/3692842249 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab380a64a0 con 0x7fab40139be0 2024-09-11T05:18:27.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:27.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 1 -- 172.21.15.138:0/3692842249 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380ac040 con 0x7fab40139be0 2024-09-11T05:18:27.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 5 monclient: authenticate success, global_id 84233 2024-09-11T05:18:27.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 10 monclient: _renew_subs 2024-09-11T05:18:27.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:27.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab4013da90 con 0x7fab40139be0 2024-09-11T05:18:27.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:27.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 monclient: dump: 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:27.912 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab377fe640 4 set_mon_vals no callback set 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 10 client.84233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 10 monclient: _renew_subs 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab4011fbc0 con 0x7fab40139be0 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 1 librados: init done 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:27.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab400a0e70 con 0x7fab40139be0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 1 -- 172.21.15.138:0/3692842249 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fab380a7ae0 con 0x7fab40139be0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 1 --2- 172.21.15.138:0/3692842249 >> v2:172.21.15.190:6800/1562782189 conn(0x7fab2403ced0 0x7fab2403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 1 -- 172.21.15.138:0/3692842249 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7fab380e9d90 con 0x7fab40139be0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 10 client.84233.objecter ms_dispatch 0x7fab40000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 3 client.84233.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 3 client.84233.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab35ffb640 20 client.84233.objecter dump_active .. 0 homeless 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab457bb640 1 --2- 172.21.15.138:0/3692842249 >> v2:172.21.15.190:6800/1562782189 conn(0x7fab2403ced0 0x7fab2403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.900+0000 7fab457bb640 10 monclient: get_auth_request con 0x7fab2403ced0 auth_method 0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.904+0000 7fab35ffb640 10 client.84233.objecter ms_handle_connect 0x7fab2403ced0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.904+0000 7fab457bb640 1 --2- 172.21.15.138:0/3692842249 >> v2:172.21.15.190:6800/1562782189 conn(0x7fab2403ced0 0x7fab2403f390 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009650 tx=0x7fab3c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.904+0000 7fab35ffb640 1 -- 172.21.15.138:0/3692842249 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fab4009efa0 con 0x7fab40139be0 2024-09-11T05:18:27.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.904+0000 7fab35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:27.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:27.904+0000 7fab35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:28.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.140+0000 7fab47a1f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:18:28.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.140+0000 7fab47a1f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:28.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.140+0000 7fab47a1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:18:28.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.140+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fab4013b8f0 con 0x7fab40139be0 2024-09-11T05:18:28.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab35ffb640 1 -- 172.21.15.138:0/3692842249 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fab380a5e20 con 0x7fab40139be0 2024-09-11T05:18:28.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:28.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:28.140 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 10 librados: watch_flush enter 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 10 librados: watch_flush exit 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 20 client.84233.objecter shutdown clearing up homeless session... 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 10 client.84233.objecter successfully canceled tick 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 >> v2:172.21.15.190:6800/1562782189 conn(0x7fab2403ced0 msgr2=0x7fab2403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3692842249 >> v2:172.21.15.190:6800/1562782189 conn(0x7fab2403ced0 0x7fab2403f390 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009650 tx=0x7fab3c009190 comp rx=0 tx=0).stop 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 10 monclient: shutdown 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 msgr2=0x7fab4011ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4011ec20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fab3801cf50 tx=0x7fab38019450 comp rx=0 tx=0).stop 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 shutdown_connections 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3692842249 >> v2:172.21.15.190:6800/1562782189 conn(0x7fab2403ced0 0x7fab2403f390 unknown :-1 s=CLOSED pgs=3442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:28.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fab40139be0 0x7fab4011ec20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:28.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fab400a0a90 0x7fab4011e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:28.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 --2- 172.21.15.138:0/3692842249 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab4009dc20 0x7fab4011e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:28.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 >> 172.21.15.138:0/3692842249 conn(0x7fab40063640 msgr2=0x7fab400a4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:28.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 shutdown_connections 2024-09-11T05:18:28.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 -- 172.21.15.138:0/3692842249 wait complete. 2024-09-11T05:18:28.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:28.144+0000 7fab47a1f640 1 librados: shutdown 2024-09-11T05:18:31.159 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient: get_monmap_and_config 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient: build_initial_monmap 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient: monmap: 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:31.295130+0000 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:31.295130+0000 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:31.290 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:31.291 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:31.291 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:31.291 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:31.291 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:31.291 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:31.291 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 1 Processor -- start 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 1 -- start start 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient: init 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89941355e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8994134dd0 addr v1:172.21.15.190:6789/0 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f89941359c0 0x7f8994141700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:31.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient(hunting): picked mon.noname-b con 0x7f89941359c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8994074990 con 0x7f89941355e0 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89940a2740 con 0x7f8994134dd0 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f89940a2d00 con 0x7f89941359c0 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899b89f640 10 monclient(hunting): authenticate will time out at 1594.731272s 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8998e3a640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7f8994134dd0 legacy=0x7f8994138a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899963b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f89941355e0 0x7f8994138350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:35810/0 (socket says 172.21.15.138:35810) 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f899963b640 1 -- 172.21.15.138:0/3906666500 learned_addr learned my addr 172.21.15.138:0/3906666500 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 1 --2- 172.21.15.138:0/3906666500 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f89941359c0 0x7f8994141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient(hunting): get_auth_request con 0x7f89941359c0 auth_method 0 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:31.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient(hunting): handle_auth_done global_id 84236 payload 274 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 1 -- 172.21.15.138:0/3906666500 >> v1:172.21.15.190:6789/0 conn(0x7f8994134dd0 legacy=0x7f8994138a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 1 -- 172.21.15.138:0/3906666500 >> v1:172.21.15.138:6789/0 conn(0x7f89941355e0 legacy=0x7f8994138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient: _finish_hunting 0 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 1 monclient: found mon.noname-b 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:18:31.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.292+0000 7f8999e3c640 1 -- 172.21.15.138:0/3906666500 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8994139170 con 0x7f89941359c0 2024-09-11T05:18:31.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 -- 172.21.15.138:0/3906666500 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f89941359c0 msgr2=0x7f8994141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:31.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 --2- 172.21.15.138:0/3906666500 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f89941359c0 0x7f8994141700 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f89840097f0 tx=0x7f898401c1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5f55bcda6fe80f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:31.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 1 -- 172.21.15.138:0/3906666500 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840b0070 con 0x7f89941359c0 2024-09-11T05:18:31.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: dump: 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: _finish_auth 0 2024-09-11T05:18:31.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 1 -- 172.21.15.138:0/3906666500 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89840ab600 con 0x7f89941359c0 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 5 monclient: authenticate success, global_id 84236 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 1 -- 172.21.15.138:0/3906666500 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840ab9d0 con 0x7f89941359c0 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 4 set_mon_vals no callback set 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f898bfff640 10 monclient: dump: 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:31.298 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:31.299 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:31.299 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:31.299 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:31.299 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:31.299 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:31.299 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:31.299 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:31.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient: shutdown 2024-09-11T05:18:31.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:31.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- 172.21.15.138:0/3906666500 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f89941359c0 msgr2=0x7f8994141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 --2- 172.21.15.138:0/3906666500 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f89941359c0 0x7f8994141700 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f89840097f0 tx=0x7f898401c1c0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- 172.21.15.138:0/3906666500 shutdown_connections 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 --2- 172.21.15.138:0/3906666500 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f89941359c0 0x7f8994141700 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- 172.21.15.138:0/3906666500 >> 172.21.15.138:0/3906666500 conn(0x7f8994063640 msgr2=0x7f8994063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- 172.21.15.138:0/3906666500 shutdown_connections 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- 172.21.15.138:0/3906666500 wait complete. 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient: build_initial_monmap 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient: monmap: 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:31.304 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 librados: starting msgr at 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 librados: starting objecter 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 Processor -- start 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- start start 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 librados: setting wanted keys 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 librados: calling monclient init 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient: init 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:31.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8994134dd0 0x7f899412b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient(hunting): picked mon.noname-c con 0x7f8994134dd0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89941355e0 0x7f899412bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient(hunting): picked mon.noname-b con 0x7f89941355e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89941359c0 0x7f899412c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient(hunting): picked mon.noname-a con 0x7f89941359c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f899406e500 con 0x7f89941359c0 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f899406e380 con 0x7f89941355e0 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8994144460 con 0x7f8994134dd0 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:31.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899b89f640 10 monclient(hunting): authenticate will time out at 1594.738369s 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8994134dd0 0x7f899412b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89941359c0 0x7f899412c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8998e3a640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89941355e0 msgr2=0x7f899412bda0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8994134dd0 0x7f899412b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:51058/0 (socket says 172.21.15.138:51058) 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 1 -- 172.21.15.138:0/2377184826 learned_addr learned my addr 172.21.15.138:0/2377184826 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 10 monclient(hunting): get_auth_request con 0x7f8994134dd0 auth_method 0 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8998e3a640 1 --2- 172.21.15.138:0/2377184826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89941355e0 0x7f899412bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient(hunting): get_auth_request con 0x7f89941359c0 auth_method 0 2024-09-11T05:18:31.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f899963b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient(hunting): handle_auth_done global_id 64560 payload 824 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 -- 172.21.15.138:0/2377184826 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8994134dd0 msgr2=0x7f899412b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 --2- 172.21.15.138:0/2377184826 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8994134dd0 0x7f899412b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 -- 172.21.15.138:0/2377184826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89941355e0 msgr2=0x7f899412bda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 --2- 172.21.15.138:0/2377184826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89941355e0 0x7f899412bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient: _finish_hunting 0 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 monclient: found mon.noname-a 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.300+0000 7f8999e3c640 1 -- 172.21.15.138:0/2377184826 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89941323e0 con 0x7f89941359c0 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8999e3c640 1 -- 172.21.15.138:0/2377184826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89941359c0 msgr2=0x7f899412c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8999e3c640 1 --2- 172.21.15.138:0/2377184826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89941359c0 0x7f899412c2e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f89840ab000 tx=0x7f898401aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 client.?.objecter ms_handle_connect 0x7f89941359c0 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 1 -- 172.21.15.138:0/2377184826 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840b0050 con 0x7f89941359c0 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: dump: 2024-09-11T05:18:31.309 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: _finish_auth 0 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 1 -- 172.21.15.138:0/2377184826 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89840bc070 con 0x7f89941359c0 2024-09-11T05:18:31.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 5 monclient: authenticate success, global_id 64560 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 10 monclient: _renew_subs 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f899412ded0 con 0x7f89941359c0 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 1 -- 172.21.15.138:0/2377184826 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89840b5c40 con 0x7f89941359c0 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 monclient: dump: 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:31.311 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 10 client.64560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 10 monclient: _renew_subs 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f898b7fe640 4 set_mon_vals no callback set 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f899412cf70 con 0x7f89941359c0 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 1 librados: init done 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:31.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8994138350 con 0x7f89941359c0 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 1 -- 172.21.15.138:0/2377184826 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f89840c1460 con 0x7f89941359c0 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 1 --2- 172.21.15.138:0/2377184826 >> v2:172.21.15.190:6800/1562782189 conn(0x7f898003ce50 0x7f898003f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 1 -- 172.21.15.138:0/2377184826 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f89840ea430 con 0x7f89941359c0 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 10 client.64560.objecter ms_dispatch 0x7f8994000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 3 client.64560.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 3 client.64560.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f8989ffb640 20 client.64560.objecter dump_active .. 0 homeless 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899963b640 1 --2- 172.21.15.138:0/2377184826 >> v2:172.21.15.190:6800/1562782189 conn(0x7f898003ce50 0x7f898003f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:31.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.304+0000 7f899963b640 10 monclient: get_auth_request con 0x7f898003ce50 auth_method 0 2024-09-11T05:18:31.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.308+0000 7f8989ffb640 10 client.64560.objecter ms_handle_connect 0x7f898003ce50 2024-09-11T05:18:31.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.308+0000 7f899963b640 1 --2- 172.21.15.138:0/2377184826 >> v2:172.21.15.190:6800/1562782189 conn(0x7f898003ce50 0x7f898003f310 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f8990007d60 tx=0x7f8990007cb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:31.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.308+0000 7f8989ffb640 1 -- 172.21.15.138:0/2377184826 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f89940a2740 con 0x7f89941359c0 2024-09-11T05:18:31.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.308+0000 7f8989ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:31.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.308+0000 7f8989ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:31.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:18:31.540 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:31.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:31.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f899409def0 con 0x7f89941359c0 2024-09-11T05:18:31.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f8989ffb640 1 -- 172.21.15.138:0/2377184826 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f89840c1ca0 con 0x7f89941359c0 2024-09-11T05:18:31.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f8989ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:31.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f8989ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:31.542 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 10 librados: watch_flush enter 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 10 librados: watch_flush exit 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 20 client.64560.objecter shutdown clearing up homeless session... 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 10 client.64560.objecter successfully canceled tick 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 >> v2:172.21.15.190:6800/1562782189 conn(0x7f898003ce50 msgr2=0x7f898003f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 --2- 172.21.15.138:0/2377184826 >> v2:172.21.15.190:6800/1562782189 conn(0x7f898003ce50 0x7f898003f310 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f8990007d60 tx=0x7f8990007cb0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 10 monclient: shutdown 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89941359c0 msgr2=0x7f899412c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:31.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 --2- 172.21.15.138:0/2377184826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89941359c0 0x7f899412c2e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f89840ab000 tx=0x7f898401aa00 comp rx=0 tx=0).stop 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 shutdown_connections 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 --2- 172.21.15.138:0/2377184826 >> v2:172.21.15.190:6800/1562782189 conn(0x7f898003ce50 0x7f898003f310 unknown :-1 s=CLOSED pgs=3443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 --2- 172.21.15.138:0/2377184826 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f89941359c0 0x7f899412c2e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 --2- 172.21.15.138:0/2377184826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89941355e0 0x7f899412bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 --2- 172.21.15.138:0/2377184826 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8994134dd0 0x7f899412b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 >> 172.21.15.138:0/2377184826 conn(0x7f8994063640 msgr2=0x7f89941338c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 shutdown_connections 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 -- 172.21.15.138:0/2377184826 wait complete. 2024-09-11T05:18:31.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:31.544+0000 7f899b89f640 1 librados: shutdown 2024-09-11T05:18:34.562 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:18:34.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient: get_monmap_and_config 2024-09-11T05:18:34.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient: build_initial_monmap 2024-09-11T05:18:34.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient: monmap: 2024-09-11T05:18:34.690 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:34.690 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:34.690 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:34.695426+0000 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:34.695426+0000 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:34.691 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:34.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 1 Processor -- start 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 1 -- start start 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient: init 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 0x7fd4d00a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient(hunting): picked mon.noname-c con 0x7fd4d00a5b40 addr v2:172.21.15.190:3300/0 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient(hunting): picked mon.noname-b con 0x7fd4d00a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient(hunting): picked mon.noname-a con 0x7fd4d009d840 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d009e110 con 0x7fd4d009d840 2024-09-11T05:18:34.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d00a69a0 con 0x7fd4d00a5b40 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d009d490 con 0x7fd4d00a7130 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4d53b6640 10 monclient(hunting): authenticate will time out at 1598.131717s 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4ceffd640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 msgr2=0x7fd4d00a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4ceffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 0x7fd4d00a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4ce7fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4ce7fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:51062/0 (socket says 172.21.15.138:51062) 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 learned_addr learned my addr 172.21.15.138:0/3511390086 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request con 0x7fd4d00a7130 auth_method 0 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:34.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.692+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fd4d009d840 auth_method 0 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:34.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.696+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:34.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.896+0000 7fd4ceffd640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 msgr2=0x7fd4d00a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:34.892 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.896+0000 7fd4ceffd640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 0x7fd4d00a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:18:34.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:34.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request con 0x7fd4d00a7130 auth_method 0 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fd4d009d840 auth_method 0 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:18:34.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:34.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:18:34.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:34.900+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:18:35.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.300+0000 7fd4ceffd640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 msgr2=0x7fd4d00a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:35.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.300+0000 7fd4ceffd640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 0x7fd4d00a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:18:35.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request con 0x7fd4d00a7130 auth_method 0 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fd4d009d840 auth_method 0 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:35.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:18:35.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:18:35.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:35.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:18:35.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:35.304+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:18:36.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.104+0000 7fd4ceffd640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 msgr2=0x7fd4d00a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:36.101 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.104+0000 7fd4ceffd640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 0x7fd4d00a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:18:36.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:36.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:36.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request con 0x7fd4d00a7130 auth_method 0 2024-09-11T05:18:36.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:36.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:36.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:36.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fd4d009d840 auth_method 0 2024-09-11T05:18:36.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:18:36.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:36.108+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:18:37.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 10 monclient(hunting): tick 2024-09-11T05:18:37.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:18:37.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 1 monclient(hunting): continuing hunt 2024-09-11T05:18:37.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 msgr2=0x7fd4d00a6460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4d00a7130 0x7fd4d00a6460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 msgr2=0x7fd4d00a5f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.190:3300/0 conn(0x7fd4d00a5b40 0x7fd4d00a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 1 -- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 msgr2=0x7fd4d0138900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 1 --2- 172.21.15.138:0/3511390086 >> v2:172.21.15.138:3300/0 conn(0x7fd4d009d840 0x7fd4d0138900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 10 monclient: _add_conns ranks=[0,1] 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.692+0000 7fd4ccff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4bc0036d0 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.696+0000 7fd4ccff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4bc007400 addr v1:172.21.15.190:6789/0 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.696+0000 7fd4ccff9640 1 -- 172.21.15.138:0/3511390086 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4d0070580 con 0x7fd4bc0036d0 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.696+0000 7fd4ccff9640 1 -- 172.21.15.138:0/3511390086 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4bc00c140 con 0x7fd4bc007400 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.696+0000 7fd4ccff9640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:37.693 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.696+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> v1:172.21.15.190:6789/0 conn(0x7fd4bc007400 legacy=0x7fd4bc0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:18:37.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.696+0000 7fd4ceffd640 1 -- 172.21.15.138:0/3511390086 >> v1:172.21.15.138:6789/0 conn(0x7fd4bc0036d0 legacy=0x7fd4bc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:37.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:37.900+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> v1:172.21.15.190:6789/0 conn(0x7fd4bc007400 legacy=0x7fd4bc0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:18:38.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:38.304+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> v1:172.21.15.190:6789/0 conn(0x7fd4bc007400 legacy=0x7fd4bc0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:18:39.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.108+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> v1:172.21.15.190:6789/0 conn(0x7fd4bc007400 legacy=0x7fd4bc0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:18:39.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.160+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135490150 0 0) 0x7fd4d0070580 con 0x7fd4bc0036d0 2024-09-11T05:18:39.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.160+0000 7fd4cdffb640 10 monclient(hunting): _negotiate 2024-09-11T05:18:39.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.160+0000 7fd4cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.160+0000 7fd4cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.160+0000 7fd4cdffb640 10 monclient(hunting): my global_id is 64563 2024-09-11T05:18:39.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.160+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4b8003530 con 0x7fd4bc0036d0 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 819143047 0 0) 0x7fd4b8003530 con 0x7fd4bc0036d0 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 >> v1:172.21.15.190:6789/0 conn(0x7fd4bc007400 legacy=0x7fd4bc0098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: _finish_hunting 0 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4bc00d3a0 con 0x7fd4bc0036d0 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: _finish_auth 0 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: _check_auth_tickets 2024-09-11T05:18:39.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7fd4c4004510 con 0x7fd4bc0036d0 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4d53b6640 5 monclient: authenticate success, global_id 64563 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4d53b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: dump: 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.169 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 >> v1:172.21.15.138:6789/0 conn(0x7fd4bc0036d0 legacy=0x7fd4bc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 reap_dead start 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient(hunting): picked mon.c con 0x7fd4b80039f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 0x7fd4b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient(hunting): picked mon.b con 0x7fd4b8004810 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient(hunting): picked mon.a con 0x7fd4b80053d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:39.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b800c230 con 0x7fd4b80053d0 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b800c3e0 con 0x7fd4b8004810 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b800c5e0 con 0x7fd4b80039f0 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cdffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4d53b6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4ce7fc640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 msgr2=0x7fd4b8004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4ce7fc640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 0x7fd4b8004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fd4b80053d0 auth_method 0 2024-09-11T05:18:39.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4ceffd640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4ceffd640 10 monclient(hunting): get_auth_request con 0x7fd4b80039f0 auth_method 0 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4ceffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_done global_id 64563 payload 293 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 msgr2=0x7fd4b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 msgr2=0x7fd4b8004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:39.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 0x7fd4b8004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 1 monclient: found mon.a 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b8003530 con 0x7fd4b80053d0 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient: _finish_auth 0 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:39.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.168+0000 7fd4ceffd640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4b8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 msgr2=0x7fd4b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4b800b9d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00a8ff0 tx=0x7fd4c801d1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dcda059567f8d30 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c80b3070 con 0x7fd4b80053d0 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 10 monclient: dump: 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.175 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c80ae410 con 0x7fd4b80053d0 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 4 set_mon_vals no callback set 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 1 -- 172.21.15.138:0/3511390086 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8003e40 con 0x7fd4b80053d0 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4cdffb640 10 monclient: dump: 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 10 monclient: shutdown 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 -- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 msgr2=0x7fd4b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4b800b9d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00a8ff0 tx=0x7fd4c801d1a0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 -- 172.21.15.138:0/3511390086 shutdown_connections 2024-09-11T05:18:39.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4b800b9d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 0x7fd4b8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/3511390086 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4b8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 -- 172.21.15.138:0/3511390086 >> 172.21.15.138:0/3511390086 conn(0x7fd4d00aba10 msgr2=0x7fd4d00a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:39.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 -- 172.21.15.138:0/3511390086 shutdown_connections 2024-09-11T05:18:39.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.176+0000 7fd4d53b6640 1 -- 172.21.15.138:0/3511390086 wait complete. 2024-09-11T05:18:39.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient: build_initial_monmap 2024-09-11T05:18:39.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient: monmap: 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 librados: starting msgr at 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 librados: starting objecter 2024-09-11T05:18:39.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 Processor -- start 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 -- start start 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 librados: setting wanted keys 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 librados: calling monclient init 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient: init 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4d01ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient(hunting): picked mon.noname-c con 0x7fd4b80039f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 0x7fd4d0138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient(hunting): picked mon.noname-b con 0x7fd4b8004810 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4d0139010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient(hunting): picked mon.noname-a con 0x7fd4b80053d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d00a8730 con 0x7fd4b80053d0 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d00a85b0 con 0x7fd4b8004810 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d00a5f20 con 0x7fd4b80039f0 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4d53b6640 10 monclient(hunting): authenticate will time out at 1602.617467s 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ceffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4d01ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ce7fc640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 msgr2=0x7fd4d0138ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4d0139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ce7fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 0x7fd4d0138ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4d0139010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:58712/0 (socket says 172.21.15.138:58712) 2024-09-11T05:18:39.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/1516125440 learned_addr learned my addr 172.21.15.138:0/1516125440 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request con 0x7fd4b80053d0 auth_method 0 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ceffd640 10 monclient(hunting): get_auth_request con 0x7fd4b80039f0 auth_method 0 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient(hunting): handle_auth_done global_id 64566 payload 824 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/1516125440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 msgr2=0x7fd4d01ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/1516125440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4d01ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/1516125440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 msgr2=0x7fd4d0138ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/1516125440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 0x7fd4d0138ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 monclient: found mon.noname-a 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.180+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/1516125440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d00a2330 con 0x7fd4b80053d0 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4cf7fe640 1 -- 172.21.15.138:0/1516125440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 msgr2=0x7fd4d0139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4cf7fe640 1 --2- 172.21.15.138:0/1516125440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4d0139010 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80b1040 tx=0x7fd4c8019f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 client.?.objecter ms_handle_connect 0x7fd4b80053d0 2024-09-11T05:18:39.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 1 -- 172.21.15.138:0/1516125440 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c80b3040 con 0x7fd4b80053d0 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: dump: 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: _finish_auth 0 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: _check_auth_tickets 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 1 -- 172.21.15.138:0/1516125440 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c80be070 con 0x7fd4b80053d0 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 5 monclient: authenticate success, global_id 64566 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 10 monclient: _renew_subs 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d0139550 con 0x7fd4b80053d0 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 1 -- 172.21.15.138:0/1516125440 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8003e40 con 0x7fd4b80053d0 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 monclient: dump: 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 10 client.64566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:39.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 10 monclient: _renew_subs 2024-09-11T05:18:39.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d01294c0 con 0x7fd4b80053d0 2024-09-11T05:18:39.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4cd7fa640 4 set_mon_vals no callback set 2024-09-11T05:18:39.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 1 librados: init done 2024-09-11T05:18:39.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:39.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:39.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 1 -- 172.21.15.138:0/1516125440 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fd4c80b87e0 con 0x7fd4b80053d0 2024-09-11T05:18:39.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d00a62b0 con 0x7fd4b80053d0 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 1 --2- 172.21.15.138:0/1516125440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4bc03d7f0 0x7fd4bc03fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 1 -- 172.21.15.138:0/1516125440 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7fd4c80eadf0 con 0x7fd4b80053d0 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 10 client.64566.objecter ms_dispatch 0x7fd4d0000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 3 client.64566.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 3 client.64566.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4b77fe640 20 client.64566.objecter dump_active .. 0 homeless 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4ceffd640 1 --2- 172.21.15.138:0/1516125440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4bc03d7f0 0x7fd4bc03fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.184+0000 7fd4ceffd640 10 monclient: get_auth_request con 0x7fd4bc03d7f0 auth_method 0 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.188+0000 7fd4ceffd640 1 --2- 172.21.15.138:0/1516125440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4bc03d7f0 0x7fd4bc03fcb0 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40046c0 tx=0x7fd4c4003970 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.188+0000 7fd4b77fe640 10 client.64566.objecter ms_handle_connect 0x7fd4bc03d7f0 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.188+0000 7fd4b77fe640 1 -- 172.21.15.138:0/1516125440 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fd4c80ee030 con 0x7fd4b80053d0 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.188+0000 7fd4b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:39.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.188+0000 7fd4b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:39.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:18:39.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:39.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd4d00a53d0 con 0x7fd4b80053d0 2024-09-11T05:18:39.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4b77fe640 1 -- 172.21.15.138:0/1516125440 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fd4c8016f40 con 0x7fd4b80053d0 2024-09-11T05:18:39.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:18:39.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4b77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:39.422 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:18:39.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 10 librados: watch_flush enter 2024-09-11T05:18:39.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 10 librados: watch_flush exit 2024-09-11T05:18:39.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 20 client.64566.objecter shutdown clearing up homeless session... 2024-09-11T05:18:39.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 10 client.64566.objecter successfully canceled tick 2024-09-11T05:18:39.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4bc03d7f0 msgr2=0x7fd4bc03fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/1516125440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4bc03d7f0 0x7fd4bc03fcb0 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40046c0 tx=0x7fd4c4003970 comp rx=0 tx=0).stop 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 10 monclient: shutdown 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 msgr2=0x7fd4d0139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.424+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/1516125440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4d0139010 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80b1040 tx=0x7fd4c8019f90 comp rx=0 tx=0).stop 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 shutdown_connections 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/1516125440 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd4bc03d7f0 0x7fd4bc03fcb0 unknown :-1 s=CLOSED pgs=3444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/1516125440 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd4b80053d0 0x7fd4d0139010 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/1516125440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd4b8004810 0x7fd4d0138ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 --2- 172.21.15.138:0/1516125440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd4b80039f0 0x7fd4d01ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 >> 172.21.15.138:0/1516125440 conn(0x7fd4d00aba10 msgr2=0x7fd4d00a4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 shutdown_connections 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 -- 172.21.15.138:0/1516125440 wait complete. 2024-09-11T05:18:39.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.428+0000 7fd4d53b6640 1 librados: shutdown 2024-09-11T05:18:39.439 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:18:39.439 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T05:18:39.439 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:18:39.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient: get_monmap_and_config 2024-09-11T05:18:39.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient: build_initial_monmap 2024-09-11T05:18:39.548 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient: monmap: 2024-09-11T05:18:39.548 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:39.548 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.548 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:39.553522+0000 2024-09-11T05:18:39.548 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:39.553522+0000 2024-09-11T05:18:39.548 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:39.549 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.549 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:39.549 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:39.549 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:39.549 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:39.549 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:39.549 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 1 Processor -- start 2024-09-11T05:18:39.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 1 -- start start 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient: init 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0ff4070010 0x7f0ff4135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff4070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ff409e6e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409eb10 0x7f0ff41389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff409eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff40a8990 con 0x7f0ff409e6e0 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f0ff40703f0 con 0x7f0ff4070010 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff409cbf0 con 0x7f0ff409eb10 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffbfcc640 10 monclient(hunting): authenticate will time out at 1602.989537s 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0ff4070010 0x7f0ff4135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f0ff4070010 0x7f0ff4135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:58728/0 (socket says 172.21.15.138:58728) 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9567640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7f0ff409e6e0 legacy=0x7f0ff41382c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 1 -- 172.21.15.138:0/1248615554 learned_addr learned my addr 172.21.15.138:0/1248615554 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient(hunting): get_auth_request con 0x7f0ff4070010 auth_method 0 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffa569640 1 --2- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409eb10 0x7f0ff41389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffa569640 10 monclient(hunting): get_auth_request con 0x7f0ff409eb10 auth_method 0 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffa569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffa569640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ffa569640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_done global_id 64569 payload 274 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 1 -- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409eb10 msgr2=0x7f0ff41389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 1 --2- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409eb10 0x7f0ff41389d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 1 -- 172.21.15.138:0/1248615554 >> v1:172.21.15.190:6789/0 conn(0x7f0ff409e6e0 legacy=0x7f0ff41382c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient: _finish_hunting 0 2024-09-11T05:18:39.553 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 1 monclient: found mon.noname-a 2024-09-11T05:18:39.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:39.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.552+0000 7f0ff9d68640 1 -- 172.21.15.138:0/1248615554 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff409ced0 con 0x7f0ff4070010 2024-09-11T05:18:39.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff9d68640 1 -- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff4070010 msgr2=0x7f0ff4135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.554 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff9d68640 1 --2- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff4070010 0x7f0ff4135450 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009e50 tx=0x7f0ff001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9b86d226f3c1d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 1 -- 172.21.15.138:0/1248615554 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff00a5720 con 0x7f0ff4070010 2024-09-11T05:18:39.555 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: dump: 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: _finish_auth 0 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: _check_auth_tickets 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 5 monclient: authenticate success, global_id 64569 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 1 -- 172.21.15.138:0/1248615554 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff00a58c0 con 0x7f0ff4070010 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 1 -- 172.21.15.138:0/1248615554 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff00a5c90 con 0x7f0ff4070010 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 4 set_mon_vals no callback set 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ff8d66640 10 monclient: dump: 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.557 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 10 monclient: shutdown 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff4070010 msgr2=0x7f0ff4135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 1 --2- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff4070010 0x7f0ff4135450 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009e50 tx=0x7f0ff001c8c0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/1248615554 shutdown_connections 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 1 --2- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409eb10 0x7f0ff41389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 1 --2- 172.21.15.138:0/1248615554 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff4070010 0x7f0ff4135450 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/1248615554 >> 172.21.15.138:0/1248615554 conn(0x7f0ff40ab830 msgr2=0x7f0ff40a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:39.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/1248615554 shutdown_connections 2024-09-11T05:18:39.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.556+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/1248615554 wait complete. 2024-09-11T05:18:39.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient: build_initial_monmap 2024-09-11T05:18:39.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient: monmap: 2024-09-11T05:18:39.564 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:39.564 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:39.564 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:39.564 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 librados: starting msgr at 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 librados: starting objecter 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 Processor -- start 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 -- start start 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 librados: setting wanted keys 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 librados: calling monclient init 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient: init 2024-09-11T05:18:39.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ff4070010 0x7f0ff412ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff4070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409e6e0 0x7f0ff412f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff409e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff409eb10 0x7f0ff40a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff409eb10 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff4142e80 con 0x7f0ff409eb10 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff4139120 con 0x7f0ff4070010 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff4143000 con 0x7f0ff409e6e0 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffbfcc640 10 monclient(hunting): authenticate will time out at 1602.997132s 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409e6e0 0x7f0ff412f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9d68640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ff4070010 msgr2=0x7f0ff412ec20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff409eb10 0x7f0ff40a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409e6e0 0x7f0ff412f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:46484/0 (socket says 172.21.15.138:46484) 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 1 -- 172.21.15.138:0/381075301 learned_addr learned my addr 172.21.15.138:0/381075301 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 10 monclient(hunting): get_auth_request con 0x7f0ff409e6e0 auth_method 0 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9d68640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ff4070010 0x7f0ff412ec20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient(hunting): get_auth_request con 0x7f0ff409eb10 auth_method 0 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient(hunting): handle_auth_done global_id 64572 payload 824 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409e6e0 msgr2=0x7f0ff412f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409e6e0 0x7f0ff412f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ff4070010 msgr2=0x7f0ff412ec20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ff4070010 0x7f0ff412ec20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient: _finish_hunting 0 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 monclient: found mon.noname-a 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 -- 172.21.15.138:0/381075301 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff412f6a0 con 0x7f0ff409eb10 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ff9567640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f0ff409e6e0 0x7f0ff412f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff409eb10 msgr2=0x7f0ff40a0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0ffa569640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff409eb10 0x7f0ff40a0440 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00d530 tx=0x7f0fec00d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0fea7fc640 10 client.?.objecter ms_handle_connect 0x7f0ff409eb10 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0fea7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fec012800 con 0x7f0ff409eb10 2024-09-11T05:18:39.569 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.560+0000 7f0fea7fc640 10 monclient: dump: 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: _finish_auth 0 2024-09-11T05:18:39.570 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fec019070 con 0x7f0ff409eb10 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 5 monclient: authenticate success, global_id 64572 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 10 monclient: _renew_subs 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff40a3e80 con 0x7f0ff409eb10 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fec012be0 con 0x7f0ff409eb10 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:39.571 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: dump: 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0febfff640 4 set_mon_vals no callback set 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 10 client.64572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 10 monclient: _renew_subs 2024-09-11T05:18:39.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff41330d0 con 0x7f0ff409eb10 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 1 librados: init done 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff40a0980 0x7f0ff40a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff40a44a0 con 0x7f0ff40a0980 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff40a0980 0x7f0ff40a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 10 monclient: get_auth_request con 0x7f0ff40a0980 auth_method 0 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_done global_id 64575 payload 824 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 1 -- 172.21.15.138:0/381075301 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ff4135450 con 0x7f0ff40a0980 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f0fec013070 con 0x7f0ff409eb10 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 --2- 172.21.15.138:0/381075301 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fd803cea0 0x7f0fd803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f0fec04f9c0 con 0x7f0ff409eb10 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 client.64572.objecter ms_dispatch 0x7f0ff4000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 3 client.64572.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 3 client.64572.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 20 client.64572.objecter dump_active .. 0 homeless 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9567640 1 --2- 172.21.15.138:0/381075301 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fd803cea0 0x7f0fd803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9567640 10 monclient: get_auth_request con 0x7f0fd803cea0 auth_method 0 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 client.64572.objecter ms_handle_connect 0x7f0ff40a0980 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff40a0980 msgr2=0x7f0ff40a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 client.64572.objecter resend_mon_ops 2024-09-11T05:18:39.574 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0ff9d68640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff40a0980 0x7f0ff40a0d60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00ae000 tx=0x7f0ff0019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff00bd020 con 0x7f0ff40a0980 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: dump: 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.575 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ff00b9880 con 0x7f0ff40a0980 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff40a0980 msgr2=0x7f0ff40a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff40a0980 0x7f0ff40a0d60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00ae000 tx=0x7f0ff0019660 comp rx=0 tx=0).stop 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff40a0980 msgr2=0x7f0ff40a0d60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.564+0000 7f0fea7fc640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff40a0980 0x7f0ff40a0d60 secure :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00ae000 tx=0x7f0ff0019660 comp rx=0 tx=0).stop 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.568+0000 7f0fea7fc640 10 client.64572.objecter ms_handle_connect 0x7f0fd803cea0 2024-09-11T05:18:39.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.568+0000 7f0ff9567640 1 --2- 172.21.15.138:0/381075301 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fd803cea0 0x7f0fd803f360 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40a5760 tx=0x7f0fe4003b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ffbfcc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ffbfcc640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff412fa80 0x7f0ff412fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ffbfcc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff40a12a0 con 0x7f0ff412fa80 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff412fa80 0x7f0ff412fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 10 monclient: get_auth_request con 0x7f0ff412fa80 auth_method 0 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 10 monclient(hunting): handle_auth_done global_id 64578 payload 824 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.572+0000 7f0ff9d68640 1 -- 172.21.15.138:0/381075301 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ff40a27c0 con 0x7f0ff412fa80 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ff9d68640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff412fa80 msgr2=0x7f0ff412fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 10 client.64572.objecter ms_handle_connect 0x7f0ff412fa80 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 10 client.64572.objecter resend_mon_ops 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ff9d68640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff412fa80 0x7f0ff412fe90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009e50 tx=0x7f0ff00a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff00bd040 con 0x7f0ff412fa80 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 10 monclient: dump: 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.579 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f0ff00039a0 con 0x7f0ff412fa80 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff412fa80 msgr2=0x7f0ff412fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff412fa80 0x7f0ff412fe90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009e50 tx=0x7f0ff00a7000 comp rx=0 tx=0).stop 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff412fa80 msgr2=0x7f0ff412fe90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:39.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0fea7fc640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff412fa80 0x7f0ff412fe90 secure :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009e50 tx=0x7f0ff00a7000 comp rx=0 tx=0).stop 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 44, 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 423206, 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:18:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:18:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:39.585 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:39.586 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:39.587 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:39.588 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:39.589 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:18:39.590 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:18:39.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.592 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.593 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:18:39.594 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:39.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 10 librados: watch_flush enter 2024-09-11T05:18:39.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 10 librados: watch_flush exit 2024-09-11T05:18:39.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 20 client.64572.objecter shutdown clearing up homeless session... 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 10 client.64572.objecter successfully canceled tick 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fd803cea0 msgr2=0x7f0fd803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 --2- 172.21.15.138:0/381075301 >> v2:172.21.15.190:6800/1562782189 conn(0x7f0fd803cea0 0x7f0fd803f360 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40a5760 tx=0x7f0fe4003b50 comp rx=0 tx=0).stop 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 10 monclient: shutdown 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff409eb10 msgr2=0x7f0ff40a0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 --2- 172.21.15.138:0/381075301 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f0ff409eb10 0x7f0ff40a0440 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00d530 tx=0x7f0fec00d9f0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffa569640 1 -- 172.21.15.138:0/381075301 reap_dead start 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffa569640 1 -- 172.21.15.138:0/381075301 reap_dead start 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 shutdown_connections 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 >> 172.21.15.138:0/381075301 conn(0x7f0ff40ab830 msgr2=0x7f0ff406efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 shutdown_connections 2024-09-11T05:18:39.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 -- 172.21.15.138:0/381075301 wait complete. 2024-09-11T05:18:39.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.576+0000 7f0ffbfcc640 1 librados: shutdown 2024-09-11T05:18:39.597 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:18:39.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T05:18:39.597 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.700+0000 7f48477b3640 10 monclient: get_monmap_and_config 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.700+0000 7f48477b3640 10 monclient: build_initial_monmap 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.700+0000 7f48477b3640 10 monclient: monmap: 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:18:39.705278+0000 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:18:39.705278+0000 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:18:39.700 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:18:39.701 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:18:39.701 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:18:39.701 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 1 Processor -- start 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 1 -- start start 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient: init 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009da20 0x7f48400a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient(hunting): picked mon.noname-b con 0x7f484009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48400a1e60 addr v1:172.21.15.138:6789/0 2024-09-11T05:18:39.703 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f48400a22a0 0x7f4840136910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient(hunting): picked mon.noname-c con 0x7f48400a22a0 addr v2:172.21.15.190:3300/0 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4840064590 con 0x7f48400a1e60 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f484009de00 con 0x7f48400a22a0 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f48400a2c70 con 0x7f484009da20 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f48477b3640 10 monclient(hunting): authenticate will time out at 1603.141314s 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009da20 0x7f48400a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009da20 0x7f48400a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:46498/0 (socket says 172.21.15.138:46498) 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 1 -- 172.21.15.138:0/4132939105 learned_addr learned my addr 172.21.15.138:0/4132939105 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient(hunting): get_auth_request con 0x7f484009da20 auth_method 0 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.704 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f4845d50640 1 -- 172.21.15.138:0/4132939105 >> v2:172.21.15.190:3300/0 conn(0x7f48400a22a0 msgr2=0x7f4840136910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f4845d50640 1 --2- 172.21.15.138:0/4132939105 >> v2:172.21.15.190:3300/0 conn(0x7f48400a22a0 0x7f4840136910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient(hunting): handle_auth_done global_id 84251 payload 274 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 1 -- 172.21.15.138:0/4132939105 >> v2:172.21.15.190:3300/0 conn(0x7f48400a22a0 msgr2=0x7f4840136910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 1 --2- 172.21.15.138:0/4132939105 >> v2:172.21.15.190:3300/0 conn(0x7f48400a22a0 0x7f4840136910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 1 -- 172.21.15.138:0/4132939105 >> v1:172.21.15.138:6789/0 conn(0x7f48400a1e60 legacy=0x7f4840136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient: _finish_hunting 0 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 1 monclient: found mon.noname-b 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:39.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:18:39.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.704+0000 7f484554f640 1 -- 172.21.15.138:0/4132939105 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48400a2f50 con 0x7f484009da20 2024-09-11T05:18:39.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f484554f640 1 -- 172.21.15.138:0/4132939105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009da20 msgr2=0x7f48400a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f484554f640 1 --2- 172.21.15.138:0/4132939105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009da20 0x7f48400a1920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f483c009870 tx=0x7f483c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=394c288308bf7cf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 1 -- 172.21.15.138:0/4132939105 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c0b2070 con 0x7f484009da20 2024-09-11T05:18:39.707 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: dump: 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: _finish_auth 0 2024-09-11T05:18:39.708 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: _check_auth_tickets 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 1 -- 172.21.15.138:0/4132939105 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f483c0ad680 con 0x7f484009da20 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 5 monclient: authenticate success, global_id 84251 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 10 monclient: get_monmap_and_config success 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 1 -- 172.21.15.138:0/4132939105 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c0ada50 con 0x7f484009da20 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 4 set_mon_vals no callback set 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f4837fff640 10 monclient: dump: 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.709 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 10 monclient: shutdown 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 -- 172.21.15.138:0/4132939105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009da20 msgr2=0x7f48400a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 --2- 172.21.15.138:0/4132939105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009da20 0x7f48400a1920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f483c009870 tx=0x7f483c01c440 comp rx=0 tx=0).stop 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 -- 172.21.15.138:0/4132939105 shutdown_connections 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 --2- 172.21.15.138:0/4132939105 >> v2:172.21.15.190:3300/0 conn(0x7f48400a22a0 0x7f4840136910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 --2- 172.21.15.138:0/4132939105 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009da20 0x7f48400a1920 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 -- 172.21.15.138:0/4132939105 >> 172.21.15.138:0/4132939105 conn(0x7f4840063640 msgr2=0x7f4840063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 -- 172.21.15.138:0/4132939105 shutdown_connections 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 -- 172.21.15.138:0/4132939105 wait complete. 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 10 monclient: build_initial_monmap 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 10 monclient: monmap: 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 librados: starting msgr at 2024-09-11T05:18:39.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 librados: starting objecter 2024-09-11T05:18:39.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 Processor -- start 2024-09-11T05:18:39.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 -- start start 2024-09-11T05:18:39.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 librados: setting wanted keys 2024-09-11T05:18:39.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 1 librados: calling monclient init 2024-09-11T05:18:39.712 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.708+0000 7f48477b3640 10 monclient: init 2024-09-11T05:18:39.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient: _reopen_session rank -1 2024-09-11T05:18:39.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:18:39.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f484009da20 0x7f4840073db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): picked mon.noname-a con 0x7f484009da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:18:39.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48400a1e60 0x7f484009e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): picked mon.noname-c con 0x7f48400a1e60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:18:39.717 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48400a22a0 0x7f484009e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): picked mon.noname-b con 0x7f48400a22a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4840140ee0 con 0x7f484009da20 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4840140d60 con 0x7f48400a22a0 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4840141060 con 0x7f48400a1e60 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): _renew_subs 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): authenticate will time out at 1603.148227s 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48400a1e60 0x7f484009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f484009da20 0x7f4840073db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48400a1e60 0x7f484009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:46514/0 (socket says 172.21.15.138:46514) 2024-09-11T05:18:39.718 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 1 -- 172.21.15.138:0/1625086801 learned_addr learned my addr 172.21.15.138:0/1625086801 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:18:39.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): get_auth_request con 0x7f48400a1e60 auth_method 0 2024-09-11T05:18:39.719 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient(hunting): get_auth_request con 0x7f484009da20 auth_method 0 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4845d50640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48400a22a0 msgr2=0x7f484009e960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4845d50640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48400a22a0 0x7f484009e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.720 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient(hunting): handle_auth_done global_id 64581 payload 824 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48400a1e60 msgr2=0x7f484009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48400a1e60 0x7f484009e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48400a22a0 msgr2=0x7f484009e960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48400a22a0 0x7f484009e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient: _finish_hunting 0 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 monclient: found mon.noname-a 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 -- 172.21.15.138:0/1625086801 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484009fdb0 con 0x7f484009da20 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48400a1e60 0x7f484009e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f484009da20 msgr2=0x7f4840073db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f484554f640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f484009da20 0x7f4840073db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f483c0af000 tx=0x7f483c01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 client.?.objecter ms_handle_connect 0x7f484009da20 2024-09-11T05:18:39.721 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c0bc020 con 0x7f484009da20 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: dump: 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.722 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: _finish_auth 0 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f483c0ade00 con 0x7f484009da20 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 5 monclient: authenticate success, global_id 64581 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient: _renew_subs 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4840134b80 con 0x7f484009da20 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c0b2040 con 0x7f484009da20 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4835ffb640 10 monclient: dump: 2024-09-11T05:18:39.723 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48377fe640 4 set_mon_vals no callback set 2024-09-11T05:18:39.724 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 client.64581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient: _renew_subs 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4840071b40 con 0x7f484009da20 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 librados: init done 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009eea0 0x7f484012e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f48400731c0 con 0x7f484009eea0 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009eea0 0x7f484012e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient: get_auth_request con 0x7f484009eea0 auth_method 0 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.725 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.712+0000 7f4844d4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f483c0b5030 con 0x7f484009da20 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4844d4e640 10 monclient(hunting): handle_auth_done global_id 84257 payload 824 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 --2- 172.21.15.138:0/1625086801 >> v2:172.21.15.190:6800/1562782189 conn(0x7f482403ce80 0x7f482403f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4844d4e640 1 -- 172.21.15.138:0/1625086801 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f484009e110 con 0x7f484009eea0 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(434..434 src has 269..434) ==== 5446+0+0 (secure 0 0 0) 0x7f483c0e9860 con 0x7f484009da20 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 client.64581.objecter ms_dispatch 0x7f4840000ba0 osd_map(434..434 src has 269..434) 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 3 client.64581.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 3 client.64581.objecter handle_osd_map decoding full epoch 434 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 20 client.64581.objecter dump_active .. 0 homeless 2024-09-11T05:18:39.726 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4845d50640 1 --2- 172.21.15.138:0/1625086801 >> v2:172.21.15.190:6800/1562782189 conn(0x7f482403ce80 0x7f482403f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4845d50640 10 monclient: get_auth_request con 0x7f482403ce80 auth_method 0 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 client.64581.objecter ms_handle_connect 0x7f484009eea0 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 client.64581.objecter resend_mon_ops 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4844d4e640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009eea0 msgr2=0x7f484012e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4844d4e640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009eea0 0x7f484012e9b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f483000fb10 tx=0x7f483000ff10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830018020 con 0x7f484009eea0 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 monclient: dump: 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.727 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f483000de00 con 0x7f484009eea0 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009eea0 msgr2=0x7f484012e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009eea0 0x7f484012e9b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f483000fb10 tx=0x7f483000ff10 comp rx=0 tx=0).stop 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009eea0 msgr2=0x7f484012e9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484009eea0 0x7f484012e9b0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f483000fb10 tx=0x7f483000ff10 comp rx=0 tx=0).stop 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4835ffb640 10 client.64581.objecter ms_handle_connect 0x7f482403ce80 2024-09-11T05:18:39.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.716+0000 7f4845d50640 1 --2- 172.21.15.138:0/1625086801 >> v2:172.21.15.190:6800/1562782189 conn(0x7f482403ce80 0x7f482403f340 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7f4838003cc0 tx=0x7f4838009270 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f48477b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:18:39.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f48477b3640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484012eef0 0x7f484012f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:18:39.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f48477b3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:18:39.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f48400a19a0 con 0x7f484012eef0 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484012eef0 0x7f484012f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 10 monclient: get_auth_request con 0x7f484012eef0 auth_method 0 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 10 monclient(hunting): handle_auth_done global_id 84260 payload 824 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 1 -- 172.21.15.138:0/1625086801 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4840134740 con 0x7f484012eef0 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484012eef0 msgr2=0x7f484012f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 10 client.64581.objecter ms_handle_connect 0x7f484012eef0 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4844d4e640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484012eef0 0x7f484012f300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4830019000 tx=0x7f4830003700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:18:39.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 10 client.64581.objecter resend_mon_ops 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830018030 con 0x7f484012eef0 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 10 monclient: dump: 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:18:39.732 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4542 (secure 0 0 0) 0x7f4830014b60 con 0x7f484012eef0 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484012eef0 msgr2=0x7f484012f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484012eef0 0x7f484012f300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4830019000 tx=0x7f4830003700 comp rx=0 tx=0).stop 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484012eef0 msgr2=0x7f484012f300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f4835ffb640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f484012eef0 0x7f484012f300 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4830019000 tx=0x7f4830003700 comp rx=0 tx=0).stop 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 44, 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 163494, 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:18:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:18:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:18:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:18:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:18:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:18:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:18:39.745 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f48477b3640 10 librados: watch_flush enter 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f48477b3640 10 librados: watch_flush exit 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f48477b3640 20 client.64581.objecter shutdown clearing up homeless session... 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.724+0000 7f48477b3640 10 client.64581.objecter successfully canceled tick 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 >> v2:172.21.15.190:6800/1562782189 conn(0x7f482403ce80 msgr2=0x7f482403f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 --2- 172.21.15.138:0/1625086801 >> v2:172.21.15.190:6800/1562782189 conn(0x7f482403ce80 0x7f482403f340 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7f4838003cc0 tx=0x7f4838009270 comp rx=0 tx=0).stop 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 10 monclient: shutdown 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f484009da20 msgr2=0x7f4840073db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f484009da20 0x7f4840073db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f483c0af000 tx=0x7f483c01ac80 comp rx=0 tx=0).stop 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f4845d50640 1 -- 172.21.15.138:0/1625086801 reap_dead start 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 shutdown_connections 2024-09-11T05:18:39.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 --2- 172.21.15.138:0/1625086801 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f484009da20 0x7f4840073db0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:18:39.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 >> 172.21.15.138:0/1625086801 conn(0x7f4840063640 msgr2=0x7f48400a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:18:39.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 shutdown_connections 2024-09-11T05:18:39.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 -- 172.21.15.138:0/1625086801 wait complete. 2024-09-11T05:18:39.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:18:39.728+0000 7f48477b3640 1 librados: shutdown 2024-09-11T05:18:39.748 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:18:39.748 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:18:53.042 INFO:tasks.ceph.osd.0.smithi138.stderr:2024-09-11T05:18:53.044+0000 7f2eb7859640 -1 snap_mapper.add_oid found existing snaps mapped on 101:2cf3a371:test-rados-api-smithi190-28187-97::foo:21, removing 2024-09-11T05:18:53.042 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T05:18:53.046+0000 7ff6017a3640 -1 snap_mapper.add_oid found existing snaps mapped on 101:2cf3a371:test-rados-api-smithi190-28187-97::foo:21, removing 2024-09-11T05:20:09.749 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T05:20:09.750 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T05:20:09.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T05:20:09.754 INFO:tasks.ceph.mon.b:Started 2024-09-11T05:20:09.754 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:20:09.754 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:20:09.852 INFO:tasks.ceph.mon.b.smithi190.stderr:2024-09-11T05:20:09.856+0000 7fa300b32e00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 463 -> 0 (sync_force? mon rebuild?) 2024-09-11T05:20:09.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient: get_monmap_and_config 2024-09-11T05:20:09.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient: build_initial_monmap 2024-09-11T05:20:09.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient: monmap: 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:09.887085+0000 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:09.887085+0000 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:09.882 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:09.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 1 Processor -- start 2024-09-11T05:20:09.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 1 -- start start 2024-09-11T05:20:09.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient: init 2024-09-11T05:20:09.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f75780a0a90 0x7f75780a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient(hunting): picked mon.noname-a con 0x7f75780a0a90 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f757809dc20 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757813bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient(hunting): picked mon.noname-b con 0x7f7578139be0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f757809efa0 con 0x7f757809dc20 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:09.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f75780a37b0 con 0x7f75780a0a90 2024-09-11T05:20:09.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:09.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f75780a2d80 con 0x7f7578139be0 2024-09-11T05:20:09.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.882+0000 7f757e8cb640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:09.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757e8cb640 10 monclient(hunting): authenticate will time out at 1693.324009s 2024-09-11T05:20:09.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f75780a0a90 0x7f75780a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:09.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757813bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:09.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f75780a0a90 0x7f75780a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45014/0 (socket says 172.21.15.138:45014) 2024-09-11T05:20:09.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 1 -- 172.21.15.138:0/2588851991 learned_addr learned my addr 172.21.15.138:0/2588851991 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:09.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient(hunting): get_auth_request con 0x7f75780a0a90 auth_method 0 2024-09-11T05:20:09.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:09.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:09.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:09.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 10 monclient(hunting): get_auth_request con 0x7f7578139be0 auth_method 0 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient(hunting): handle_auth_done global_id 64599 payload 274 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 1 -- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 msgr2=0x7f757813bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:09.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 1 --2- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757813bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:09.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 1 -- 172.21.15.138:0/2588851991 >> v1:172.21.15.190:6789/0 conn(0x7f757809dc20 legacy=0x7f75780a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:09.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient: _finish_hunting 0 2024-09-11T05:20:09.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 1 monclient: found mon.noname-a 2024-09-11T05:20:09.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:09.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:09.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f7577fff640 1 -- 172.21.15.138:0/2588851991 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757809ecc0 con 0x7f75780a0a90 2024-09-11T05:20:09.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.886+0000 7f757ce68640 1 --2- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757813bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 1 -- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 msgr2=0x7f75780a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 1 --2- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75780a0e70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f756800bb10 tx=0x7f756801e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6f9d9af3eaa42c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 1 -- 172.21.15.138:0/2588851991 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75680b4070 con 0x7f75780a0a90 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: dump: 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:09.897 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: _finish_auth 0 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: _check_auth_tickets 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 1 -- 172.21.15.138:0/2588851991 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75680a7960 con 0x7f75780a0a90 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 5 monclient: authenticate success, global_id 64599 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:09.898 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 1 -- 172.21.15.138:0/2588851991 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75680a7d30 con 0x7f75780a0a90 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 4 set_mon_vals no callback set 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f75777fe640 10 monclient: dump: 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:09.899 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient: shutdown 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 msgr2=0x7f75780a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 --2- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75780a0e70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f756800bb10 tx=0x7f756801e970 comp rx=0 tx=0).stop 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- 172.21.15.138:0/2588851991 shutdown_connections 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 --2- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757813bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 --2- 172.21.15.138:0/2588851991 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75780a0e70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- 172.21.15.138:0/2588851991 >> 172.21.15.138:0/2588851991 conn(0x7f7578063640 msgr2=0x7f7578063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- 172.21.15.138:0/2588851991 shutdown_connections 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- 172.21.15.138:0/2588851991 wait complete. 2024-09-11T05:20:09.900 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient: build_initial_monmap 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient: monmap: 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 librados: starting msgr at 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 librados: starting objecter 2024-09-11T05:20:09.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 Processor -- start 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- start start 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 librados: setting wanted keys 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 librados: calling monclient init 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient: init 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 0x7f75781d0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient(hunting): picked mon.noname-b con 0x7f757809dc20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75781337b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient(hunting): picked mon.noname-a con 0x7f75780a0a90 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757811d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient(hunting): picked mon.noname-c con 0x7f7578139be0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f757813ea50 con 0x7f75780a0a90 2024-09-11T05:20:09.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f757813e8d0 con 0x7f757809dc20 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f757813ebd0 con 0x7f7578139be0 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757e8cb640 10 monclient(hunting): authenticate will time out at 1693.331327s 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75781337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757811d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75781337b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45026/0 (socket says 172.21.15.138:45026) 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757811d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:40782/0 (socket says 172.21.15.138:40782) 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 -- 172.21.15.138:0/1615738728 learned_addr learned my addr 172.21.15.138:0/1615738728 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient(hunting): get_auth_request con 0x7f75780a0a90 auth_method 0 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:09.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 10 monclient(hunting): get_auth_request con 0x7f7578139be0 auth_method 0 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 0x7f75781d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 10 monclient(hunting): get_auth_request con 0x7f757809dc20 auth_method 0 2024-09-11T05:20:09.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient(hunting): handle_auth_done global_id 64602 payload 824 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 1 -- 172.21.15.138:0/1615738728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 msgr2=0x7f75781d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 -- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 msgr2=0x7f757811d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757811d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 -- 172.21.15.138:0/1615738728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 msgr2=0x7f75781d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 1 -- 172.21.15.138:0/1615738728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 msgr2=0x7f75781d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 0x7f75781d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f7577fff640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 0x7f75781d0300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 0x7f75781d0300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:20:09.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 monclient: found mon.noname-a 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f756f7fe640 1 -- 172.21.15.138:0/1615738728 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7578121170 con 0x7f75780a0a90 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.890+0000 7f757ce68640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757811d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f756f7fe640 1 -- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 msgr2=0x7f75781337b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f756f7fe640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75781337b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f756000c450 tx=0x7f756000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 client.?.objecter ms_handle_connect 0x7f75780a0a90 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 1 -- 172.21.15.138:0/1615738728 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75600119a0 con 0x7f75780a0a90 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: dump: 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:09.906 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: _finish_auth 0 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: _check_auth_tickets 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 1 -- 172.21.15.138:0/1615738728 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7560017070 con 0x7f75780a0a90 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:09.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 1 -- 172.21.15.138:0/1615738728 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560011da0 con 0x7f75780a0a90 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 5 monclient: authenticate success, global_id 64602 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 10 monclient: _renew_subs 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757809e110 con 0x7f75780a0a90 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f7576ffd640 4 set_mon_vals no callback set 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 monclient: dump: 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:09.908 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 10 client.64602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 10 monclient: _renew_subs 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75780a5320 con 0x7f75780a0a90 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 1 librados: init done 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:09.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75780a0e70 con 0x7f75780a0a90 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 1 -- 172.21.15.138:0/1615738728 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f756001c460 con 0x7f75780a0a90 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 1 --2- 172.21.15.138:0/1615738728 >> v2:172.21.15.190:6800/1562782189 conn(0x7f755803ced0 0x7f755803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 1 -- 172.21.15.138:0/1615738728 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(514..514 src has 269..514) ==== 5845+0+0 (secure 0 0 0) 0x7f756004f9b0 con 0x7f75780a0a90 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 10 client.64602.objecter ms_dispatch 0x7f7578000ba0 osd_map(514..514 src has 269..514) 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 3 client.64602.objecter handle_osd_map got epochs [514,514] > 0 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 3 client.64602.objecter handle_osd_map decoding full epoch 514 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.894+0000 7f75757fa640 20 client.64602.objecter dump_active .. 0 homeless 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.898+0000 7f7577fff640 1 --2- 172.21.15.138:0/1615738728 >> v2:172.21.15.190:6800/1562782189 conn(0x7f755803ced0 0x7f755803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.898+0000 7f7577fff640 10 monclient: get_auth_request con 0x7f755803ced0 auth_method 0 2024-09-11T05:20:09.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.898+0000 7f7577fff640 1 --2- 172.21.15.138:0/1615738728 >> v2:172.21.15.190:6800/1562782189 conn(0x7f755803ced0 0x7f755803f390 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7f757806a140 tx=0x7f7568008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:09.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.898+0000 7f75757fa640 10 client.64602.objecter ms_handle_connect 0x7f755803ced0 2024-09-11T05:20:09.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.902+0000 7f75757fa640 1 -- 172.21.15.138:0/1615738728 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f756001ac60 con 0x7f75780a0a90 2024-09-11T05:20:09.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.902+0000 7f75757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:09.911 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:09.902+0000 7f75757fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:10.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.134+0000 7f757e8cb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:20:10.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.134+0000 7f757e8cb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:10.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.134+0000 7f757e8cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:10.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.134+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f757806ccd0 con 0x7f75780a0a90 2024-09-11T05:20:10.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.134+0000 7f75757fa640 1 -- 172.21.15.138:0/1615738728 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f7560010cf0 con 0x7f75780a0a90 2024-09-11T05:20:10.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.134+0000 7f75757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:10.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.134+0000 7f75757fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:10.135 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":44,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:20:10.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 10 librados: watch_flush enter 2024-09-11T05:20:10.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 10 librados: watch_flush exit 2024-09-11T05:20:10.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 20 client.64602.objecter shutdown clearing up homeless session... 2024-09-11T05:20:10.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 10 client.64602.objecter successfully canceled tick 2024-09-11T05:20:10.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 >> v2:172.21.15.190:6800/1562782189 conn(0x7f755803ced0 msgr2=0x7f755803f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 --2- 172.21.15.138:0/1615738728 >> v2:172.21.15.190:6800/1562782189 conn(0x7f755803ced0 0x7f755803f390 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7f757806a140 tx=0x7f7568008040 comp rx=0 tx=0).stop 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 10 monclient: shutdown 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 msgr2=0x7f75781337b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75781337b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f756000c450 tx=0x7f756000c910 comp rx=0 tx=0).stop 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 shutdown_connections 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 --2- 172.21.15.138:0/1615738728 >> v2:172.21.15.190:6800/1562782189 conn(0x7f755803ced0 0x7f755803f390 unknown :-1 s=CLOSED pgs=3451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f7578139be0 0x7f757811d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f75780a0a90 0x7f75781337b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 --2- 172.21.15.138:0/1615738728 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f757809dc20 0x7f75781d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 >> 172.21.15.138:0/1615738728 conn(0x7f7578063640 msgr2=0x7f757809d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 shutdown_connections 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 -- 172.21.15.138:0/1615738728 wait complete. 2024-09-11T05:20:10.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:10.138+0000 7f757e8cb640 1 librados: shutdown 2024-09-11T05:20:13.155 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:20:13.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient: get_monmap_and_config 2024-09-11T05:20:13.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient: build_initial_monmap 2024-09-11T05:20:13.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient: monmap: 2024-09-11T05:20:13.282 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:13.283 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:13.283 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:13.287326+0000 2024-09-11T05:20:13.283 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:13.287326+0000 2024-09-11T05:20:13.283 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:13.284 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:13.284 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:13.284 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:13.284 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:13.284 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:13.285 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:13.285 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:13.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 1 Processor -- start 2024-09-11T05:20:13.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 1 -- start start 2024-09-11T05:20:13.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient: init 2024-09-11T05:20:13.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:13.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-11T05:20:13.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52581355e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:20:13.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5258134dd0 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:13.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:13.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient(hunting): picked mon.noname-b con 0x7f52581359c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:13.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5258074990 con 0x7f52581355e0 2024-09-11T05:20:13.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52580a2740 con 0x7f5258134dd0 2024-09-11T05:20:13.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:13.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f52580a2d00 con 0x7f52581359c0 2024-09-11T05:20:13.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:13.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525edd7640 10 monclient(hunting): authenticate will time out at 1696.723711s 2024-09-11T05:20:13.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525cb73640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f52581355e0 0x7f5258138350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:36446/0 (socket says 172.21.15.138:36446) 2024-09-11T05:20:13.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 learned_addr learned my addr 172.21.15.138:0/3092335329 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:20:13.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.282+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:13.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525d374640 10 monclient(hunting): get_auth_request con 0x7f52581359c0 auth_method 0 2024-09-11T05:20:13.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525d374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:13.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525d374640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:13.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525d374640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:13.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:20:13.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:13.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:13.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:13.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 >> v1:172.21.15.138:6789/0 conn(0x7f52581355e0 legacy=0x7f5258138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:13.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.286+0000 7f5257fff640 1 -- 172.21.15.138:0/3092335329 >> v1:172.21.15.190:6789/0 conn(0x7f5258134dd0 legacy=0x7f5258138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:13.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:13.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 10 monclient(hunting): get_auth_request con 0x7f52581359c0 auth_method 0 2024-09-11T05:20:13.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:13.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:13.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:13.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:20:13.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:13.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:13.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.490+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:20:13.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:13.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 10 monclient(hunting): get_auth_request con 0x7f52581359c0 auth_method 0 2024-09-11T05:20:13.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:13.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:13.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:13.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:20:13.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:13.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:13.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:13.894+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:20:14.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:14.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 10 monclient(hunting): get_auth_request con 0x7f52581359c0 auth_method 0 2024-09-11T05:20:14.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:14.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:14.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:14.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:20:14.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:14.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:14.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:14.698+0000 7f525d374640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:20:16.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 10 monclient(hunting): tick 2024-09-11T05:20:16.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:20:16.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 1 monclient(hunting): continuing hunt 2024-09-11T05:20:16.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:20:16.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 msgr2=0x7f5258141700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:16.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f52581359c0 0x7f5258141700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:16.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 1 -- 172.21.15.138:0/3092335329 >> v1:172.21.15.190:6789/0 conn(0x7f5258134dd0 legacy=0x7f5258138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:16.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 1 -- 172.21.15.138:0/3092335329 >> v1:172.21.15.138:6789/0 conn(0x7f52581355e0 legacy=0x7f5258138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.282+0000 7f52567fc640 10 monclient: _add_conns ranks=[3,4] 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 10 monclient(hunting): picked mon.noname-a con 0x7f5244003800 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 10 monclient(hunting): picked mon.noname-c con 0x7f5244006370 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 1 -- 172.21.15.138:0/3092335329 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f52580a2d00 con 0x7f5244003800 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 1 -- 172.21.15.138:0/3092335329 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5244008ed0 con 0x7f5244006370 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f52567fc640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 10 monclient(hunting): get_auth_request con 0x7f5244003800 auth_method 0 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 10 monclient(hunting): get_auth_request con 0x7f5244006370 auth_method 0 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 msgr2=0x7f5244005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 msgr2=0x7f5244005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 msgr2=0x7f5244008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 msgr2=0x7f5244008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:16.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:16.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.286+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:16.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 10 monclient(hunting): get_auth_request con 0x7f5244003800 auth_method 0 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 10 monclient(hunting): get_auth_request con 0x7f5244006370 auth_method 0 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:16.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 msgr2=0x7f5244005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:20:16.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 msgr2=0x7f5244005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:16.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:16.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:20:16.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 msgr2=0x7f5244008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:20:16.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 msgr2=0x7f5244008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:16.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:16.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.490+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.138:3300/0 conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient(hunting): get_auth_request con 0x7f5244003800 auth_method 0 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 10 monclient(hunting): get_auth_request con 0x7f5244006370 auth_method 0 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:16.894 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 msgr2=0x7f5244008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 msgr2=0x7f5244008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f5257fff640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient(hunting): handle_auth_done global_id 64605 payload 274 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 msgr2=0x7f5244008810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> v2:172.21.15.190:3300/0 conn(0x7f5244006370 0x7f5244008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient: _finish_hunting 0 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 1 monclient: found mon.noname-a 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:16.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:16.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.894+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5258139170 con 0x7f5244003800 2024-09-11T05:20:16.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525cb73640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5244003800 msgr2=0x7f5244005c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:16.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525cb73640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5244003800 0x7f5244005c80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5244006340 tx=0x7f524c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc5999e2cbda748 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 1 -- 172.21.15.138:0/3092335329 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c0ab070 con 0x7f5244003800 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: dump: 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:16.902 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: _finish_auth 0 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: _check_auth_tickets 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 1 -- 172.21.15.138:0/3092335329 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f524c0a5880 con 0x7f5244003800 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 1 -- 172.21.15.138:0/3092335329 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c0a5c50 con 0x7f5244003800 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f52577fe640 10 monclient: dump: 2024-09-11T05:20:16.903 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 5 monclient: authenticate success, global_id 64605 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:16.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 4 set_mon_vals no callback set 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 10 monclient: shutdown 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 1 -- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5244003800 msgr2=0x7f5244005c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5244003800 0x7f5244005c80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5244006340 tx=0x7f524c01c910 comp rx=0 tx=0).stop 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525d374640 1 -- 172.21.15.138:0/3092335329 reap_dead start 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 1 -- 172.21.15.138:0/3092335329 shutdown_connections 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 1 --2- 172.21.15.138:0/3092335329 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5244003800 0x7f5244005c80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 1 -- 172.21.15.138:0/3092335329 >> 172.21.15.138:0/3092335329 conn(0x7f5258063640 msgr2=0x7f5258063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 1 -- 172.21.15.138:0/3092335329 shutdown_connections 2024-09-11T05:20:16.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.902+0000 7f525edd7640 1 -- 172.21.15.138:0/3092335329 wait complete. 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient: build_initial_monmap 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient: monmap: 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 librados: starting msgr at 2024-09-11T05:20:16.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 librados: starting objecter 2024-09-11T05:20:16.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 Processor -- start 2024-09-11T05:20:16.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 -- start start 2024-09-11T05:20:16.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 librados: setting wanted keys 2024-09-11T05:20:16.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 librados: calling monclient init 2024-09-11T05:20:16.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient: init 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5244003800 0x7f5258130850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient(hunting): picked mon.noname-b con 0x7f5244003800 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5258130d90 0x7f5258131210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient(hunting): picked mon.noname-c con 0x7f5258130d90 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52581d0760 0x7f52581d0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient(hunting): picked mon.noname-a con 0x7f52581d0760 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f525806e4c0 con 0x7f52581d0760 2024-09-11T05:20:16.912 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f525806e340 con 0x7f5244003800 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5258144460 con 0x7f5258130d90 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525edd7640 10 monclient(hunting): authenticate will time out at 1700.347192s 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5258130d90 0x7f5258131210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5258130d90 0x7f5258131210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:40818/0 (socket says 172.21.15.138:40818) 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 -- 172.21.15.138:0/628075692 learned_addr learned my addr 172.21.15.138:0/628075692 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient(hunting): get_auth_request con 0x7f5258130d90 auth_method 0 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:16.913 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525d374640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52581d0760 0x7f52581d0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525d374640 10 monclient(hunting): get_auth_request con 0x7f52581d0760 auth_method 0 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525d374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525d374640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525d374640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525cb73640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5244003800 0x7f5258130850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525d374640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525d374640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525d374640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525cb73640 10 monclient(hunting): get_auth_request con 0x7f5244003800 auth_method 0 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525cb73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:16.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525cb73640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f525cb73640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient(hunting): handle_auth_done global_id 84290 payload 824 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 -- 172.21.15.138:0/628075692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5244003800 msgr2=0x7f5258130850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5244003800 0x7f5258130850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 -- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52581d0760 msgr2=0x7f52581d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52581d0760 0x7f52581d0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient: _finish_hunting 0 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 monclient: found mon.noname-c 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.906+0000 7f5257fff640 1 -- 172.21.15.138:0/628075692 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5258141130 con 0x7f5258130d90 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f5257fff640 1 -- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5258130d90 msgr2=0x7f5258131210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f5257fff640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5258130d90 0x7f5258131210 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5248011600 tx=0x7f5248011ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:16.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 client.?.objecter ms_handle_connect 0x7f5258130d90 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 1 -- 172.21.15.138:0/628075692 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52480074f0 con 0x7f5258130d90 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: dump: 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:16.916 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: _finish_auth 0 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: _check_auth_tickets 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 5 monclient: authenticate success, global_id 84290 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 10 monclient: _renew_subs 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525813a940 con 0x7f5258130d90 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 10 client.84290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 10 monclient: _renew_subs 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:16.917 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f525813b1e0 con 0x7f5258130d90 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 1 librados: init done 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 1 -- 172.21.15.138:0/628075692 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f524801a070 con 0x7f5258130d90 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 1 -- 172.21.15.138:0/628075692 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248005a10 con 0x7f5258130d90 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 10 monclient: dump: 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:16.918 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f5256ffd640 4 set_mon_vals no callback set 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5258138350 con 0x7f5258130d90 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.910+0000 7f52557fa640 1 -- 172.21.15.138:0/628075692 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5248005bb0 con 0x7f5258130d90 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f52557fa640 1 --2- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 0x7f52440402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f52557fa640 1 -- 172.21.15.138:0/628075692 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(518..518 src has 269..518) ==== 5464+0+0 (secure 0 0 0) 0x7f524804fc20 con 0x7f5258130d90 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f52557fa640 10 client.84290.objecter ms_dispatch 0x7f5258000ba0 osd_map(518..518 src has 269..518) 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f52557fa640 3 client.84290.objecter handle_osd_map got epochs [518,518] > 0 2024-09-11T05:20:16.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f52557fa640 3 client.84290.objecter handle_osd_map decoding full epoch 518 2024-09-11T05:20:16.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f52557fa640 20 client.84290.objecter dump_active .. 0 homeless 2024-09-11T05:20:16.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f525cb73640 1 --2- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 0x7f52440402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:16.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f525cb73640 10 monclient: get_auth_request con 0x7f524403ddf0 auth_method 0 2024-09-11T05:20:16.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f525cb73640 1 -- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 msgr2=0x7f52440402b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:20:16.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f525cb73640 1 -- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 msgr2=0x7f52440402b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:20:16.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f525cb73640 1 --2- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 0x7f52440402b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f524c0ad000 tx=0x7f524c010040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:16.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.914+0000 7f525cb73640 1 --2- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 0x7f52440402b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:16.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.918+0000 7f52557fa640 1 -- 172.21.15.138:0/628075692 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f525813a940 con 0x7f5258130d90 2024-09-11T05:20:16.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.918+0000 7f52557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:16.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:16.918+0000 7f52557fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:17.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.118+0000 7f525cb73640 1 --2- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 0x7f52440402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.118+0000 7f525cb73640 10 monclient: get_auth_request con 0x7f524403ddf0 auth_method 0 2024-09-11T05:20:17.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.118+0000 7f525cb73640 1 --2- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 0x7f52440402b0 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f524c0ad000 tx=0x7f524c010040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.118+0000 7f52557fa640 10 client.84290.objecter ms_handle_connect 0x7f524403ddf0 2024-09-11T05:20:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f525edd7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:20:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f525edd7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f525edd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f52580a1f90 con 0x7f5258130d90 2024-09-11T05:20:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f52557fa640 1 -- 172.21.15.138:0/628075692 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f52580a2740 con 0x7f5258130d90 2024-09-11T05:20:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f52557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:17.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f52557fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:17.150 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:20:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f525edd7640 10 librados: watch_flush enter 2024-09-11T05:20:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f525edd7640 10 librados: watch_flush exit 2024-09-11T05:20:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f525edd7640 20 client.84290.objecter shutdown clearing up homeless session... 2024-09-11T05:20:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.150+0000 7f525edd7640 10 client.84290.objecter successfully canceled tick 2024-09-11T05:20:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 msgr2=0x7f52440402b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 --2- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 0x7f52440402b0 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f524c0ad000 tx=0x7f524c010040 comp rx=0 tx=0).stop 2024-09-11T05:20:17.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 10 monclient: shutdown 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5258130d90 msgr2=0x7f5258131210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5258130d90 0x7f5258131210 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5248011600 tx=0x7f5248011ac0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 shutdown_connections 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 --2- 172.21.15.138:0/628075692 >> v2:172.21.15.190:6800/1562782189 conn(0x7f524403ddf0 0x7f52440402b0 unknown :-1 s=CLOSED pgs=3453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f52581d0760 0x7f52581d0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5258130d90 0x7f5258131210 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 --2- 172.21.15.138:0/628075692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5244003800 0x7f5258130850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 >> 172.21.15.138:0/628075692 conn(0x7f5258063640 msgr2=0x7f52580a63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 shutdown_connections 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 -- 172.21.15.138:0/628075692 wait complete. 2024-09-11T05:20:17.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.154+0000 7f525edd7640 1 librados: shutdown 2024-09-11T05:20:17.167 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:20:17.167 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:20:17.167 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:20:17.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.274+0000 7fb1a5eff640 10 monclient: get_monmap_and_config 2024-09-11T05:20:17.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.274+0000 7fb1a5eff640 10 monclient: build_initial_monmap 2024-09-11T05:20:17.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.274+0000 7fb1a5eff640 10 monclient: monmap: 2024-09-11T05:20:17.274 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:17.274 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:17.280019+0000 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:17.280019+0000 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:17.275 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.277 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.274+0000 7fb1a5eff640 1 Processor -- start 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.274+0000 7fb1a5eff640 1 -- start start 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.274+0000 7fb1a5eff640 10 monclient: init 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb1a009da20 0x7fb1a00a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a009da20 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fb1a00a0a90 0x7fb1a00a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a00a0a90 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1a0134850 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a00a13e0 con 0x7fb1a0134850 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.278 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a009efa0 con 0x7fb1a00a0a90 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a009de00 con 0x7fb1a009da20 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb1a5eff640 10 monclient(hunting): authenticate will time out at 1700.716247s 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fb1a00a0a90 0x7fb1a00a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fb1a00a0a90 0x7fb1a00a0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45072/0 (socket says 172.21.15.138:45072) 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 1 -- 172.21.15.138:0/3307541200 learned_addr learned my addr 172.21.15.138:0/3307541200 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient(hunting): get_auth_request con 0x7fb1a00a0a90 auth_method 0 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19f7fe640 1 --2- 172.21.15.138:0/3307541200 >> v2:172.21.15.190:3300/0 conn(0x7fb1a009da20 0x7fb1a00a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19f7fe640 10 monclient(hunting): get_auth_request con 0x7fb1a009da20 auth_method 0 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient(hunting): handle_auth_done global_id 64611 payload 274 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 1 -- 172.21.15.138:0/3307541200 >> v2:172.21.15.190:3300/0 conn(0x7fb1a009da20 msgr2=0x7fb1a00a26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 1 --2- 172.21.15.138:0/3307541200 >> v2:172.21.15.190:3300/0 conn(0x7fb1a009da20 0x7fb1a00a26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 1 -- 172.21.15.138:0/3307541200 >> v1:172.21.15.190:6789/0 conn(0x7fb1a0134850 legacy=0x7fb1a0134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient: _finish_hunting 0 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 1 monclient: found mon.noname-a 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:17.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.278+0000 7fb19effd640 1 -- 172.21.15.138:0/3307541200 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a0064590 con 0x7fb1a00a0a90 2024-09-11T05:20:17.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19effd640 1 -- 172.21.15.138:0/3307541200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a00a0a90 msgr2=0x7fb1a00a0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19effd640 1 --2- 172.21.15.138:0/3307541200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a00a0a90 0x7fb1a00a0ea0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00a6ef0 tx=0x7fb19801e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=589a3287046f1181 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 1 -- 172.21.15.138:0/3307541200 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1980ad070 con 0x7fb1a00a0a90 2024-09-11T05:20:17.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: dump: 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:20:17.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 1 -- 172.21.15.138:0/3307541200 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb198004230 con 0x7fb1a00a0a90 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 1 -- 172.21.15.138:0/3307541200 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1980a7670 con 0x7fb1a00a0a90 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 5 monclient: authenticate success, global_id 64611 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 4 set_mon_vals no callback set 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19e7fc640 10 monclient: dump: 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.286 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient: shutdown 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- 172.21.15.138:0/3307541200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a00a0a90 msgr2=0x7fb1a00a0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 --2- 172.21.15.138:0/3307541200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a00a0a90 0x7fb1a00a0ea0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00a6ef0 tx=0x7fb19801e860 comp rx=0 tx=0).stop 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- 172.21.15.138:0/3307541200 shutdown_connections 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 --2- 172.21.15.138:0/3307541200 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a00a0a90 0x7fb1a00a0ea0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 --2- 172.21.15.138:0/3307541200 >> v2:172.21.15.190:3300/0 conn(0x7fb1a009da20 0x7fb1a00a26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- 172.21.15.138:0/3307541200 >> 172.21.15.138:0/3307541200 conn(0x7fb1a0063640 msgr2=0x7fb1a0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:17.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- 172.21.15.138:0/3307541200 shutdown_connections 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- 172.21.15.138:0/3307541200 wait complete. 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient: build_initial_monmap 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient: monmap: 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.288 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 librados: starting msgr at 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 librados: starting objecter 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 Processor -- start 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- start start 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 librados: setting wanted keys 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 librados: calling monclient init 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient: init 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1a009da20 0x7fb1a01cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1a00a0a90 0x7fb1a012fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a00a0a90 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0134850 0x7fb1a01d0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a0134850 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a013eba0 con 0x7fb1a0134850 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a013ed20 con 0x7fb1a00a0a90 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a013eea0 con 0x7fb1a009da20 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb1a5eff640 10 monclient(hunting): authenticate will time out at 1700.723053s 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0134850 0x7fb1a01d0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1a009da20 0x7fb1a01cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0134850 0x7fb1a01d0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45084/0 (socket says 172.21.15.138:45084) 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1a009da20 0x7fb1a01cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:40820/0 (socket says 172.21.15.138:40820) 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 -- 172.21.15.138:0/2168231951 learned_addr learned my addr 172.21.15.138:0/2168231951 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:17.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient(hunting): get_auth_request con 0x7fb1a0134850 auth_method 0 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 10 monclient(hunting): get_auth_request con 0x7fb1a009da20 auth_method 0 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19effd640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1a00a0a90 0x7fb1a012fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient(hunting): handle_auth_done global_id 64614 payload 824 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1a009da20 msgr2=0x7fb1a01cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1a009da20 0x7fb1a01cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1a00a0a90 msgr2=0x7fb1a012fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1a00a0a90 0x7fb1a012fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 monclient: found mon.noname-a 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19ffff640 1 -- 172.21.15.138:0/2168231951 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a011e4c0 con 0x7fb1a0134850 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19effd640 10 monclient: get_auth_request con 0x7fb1a00a0a90 auth_method 0 2024-09-11T05:20:17.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.282+0000 7fb19effd640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1a00a0a90 0x7fb1a012fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:20:17.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19ffff640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0134850 msgr2=0x7fb1a01d0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19ffff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0134850 0x7fb1a01d0750 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb18800c3c0 tx=0x7fb18800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 client.?.objecter ms_handle_connect 0x7fb1a0134850 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1880119a0 con 0x7fb1a0134850 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: dump: 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.297 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: _finish_auth 0 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: _check_auth_tickets 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 5 monclient: authenticate success, global_id 64614 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb188017070 con 0x7fb1a0134850 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 10 monclient: _renew_subs 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188011d80 con 0x7fb1a0134850 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a0074180 con 0x7fb1a0134850 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:17.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 monclient: dump: 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19dffb640 4 set_mon_vals no callback set 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 10 client.64614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 10 monclient: _renew_subs 2024-09-11T05:20:17.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a013c880 con 0x7fb1a0134850 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 1 librados: init done 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0130310 0x7fb1a01d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a00a2e10 con 0x7fb1a0130310 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0130310 0x7fb1a01d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 10 monclient: get_auth_request con 0x7fb1a0130310 auth_method 0 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_done global_id 64617 payload 824 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19f7fe640 1 -- 172.21.15.138:0/2168231951 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1a00a26c0 con 0x7fb1a0130310 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fb18801c460 con 0x7fb1a0134850 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 1 --2- 172.21.15.138:0/2168231951 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb18403ced0 0x7fb18403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(518..518 src has 269..518) ==== 5464+0+0 (secure 0 0 0) 0x7fb18804f7c0 con 0x7fb1a0134850 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 10 client.64614.objecter ms_dispatch 0x7fb1a0000ba0 osd_map(518..518 src has 269..518) 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 3 client.64614.objecter handle_osd_map got epochs [518,518] > 0 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 3 client.64614.objecter handle_osd_map decoding full epoch 518 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb197fff640 20 client.64614.objecter dump_active .. 0 homeless 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19effd640 1 --2- 172.21.15.138:0/2168231951 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb18403ced0 0x7fb18403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.286+0000 7fb19effd640 10 monclient: get_auth_request con 0x7fb18403ced0 auth_method 0 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 10 client.64614.objecter ms_handle_connect 0x7fb1a0130310 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 10 client.64614.objecter resend_mon_ops 2024-09-11T05:20:17.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb19f7fe640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0130310 msgr2=0x7fb1a01d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb19f7fe640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0130310 0x7fb1a01d0c90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb190004650 tx=0x7fb190007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190014590 con 0x7fb1a0130310 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 10 monclient: dump: 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.302 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb19001a070 con 0x7fb1a0130310 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0130310 msgr2=0x7fb1a01d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0130310 0x7fb1a01d0c90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb190004650 tx=0x7fb190007700 comp rx=0 tx=0).stop 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0130310 msgr2=0x7fb1a01d0c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0130310 0x7fb1a01d0c90 secure :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb190004650 tx=0x7fb190007700 comp rx=0 tx=0).stop 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb197fff640 10 client.64614.objecter ms_handle_connect 0x7fb18403ced0 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.290+0000 7fb19effd640 1 --2- 172.21.15.138:0/2168231951 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb18403ced0 0x7fb18403f390 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7fb1a006e190 tx=0x7fb1980a9070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb1a5eff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb1a5eff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a01d11d0 0x7fb1a00700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb1a5eff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a01d15e0 con 0x7fb1a01d11d0 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a01d11d0 0x7fb1a00700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 10 monclient: get_auth_request con 0x7fb1a01d11d0 auth_method 0 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 10 monclient(hunting): handle_auth_done global_id 64620 payload 824 2024-09-11T05:20:17.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.294+0000 7fb19f7fe640 1 -- 172.21.15.138:0/2168231951 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb1a006cca0 con 0x7fb1a01d11d0 2024-09-11T05:20:17.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 10 client.64614.objecter ms_handle_connect 0x7fb1a01d11d0 2024-09-11T05:20:17.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 10 client.64614.objecter resend_mon_ops 2024-09-11T05:20:17.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb19f7fe640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a01d11d0 msgr2=0x7fb1a00700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb19f7fe640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a01d11d0 0x7fb1a00700f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb190004650 tx=0x7fb190004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190014c90 con 0x7fb1a01d11d0 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 10 monclient: dump: 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7fb190014c70 con 0x7fb1a01d11d0 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:20:17.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a01d11d0 msgr2=0x7fb1a00700f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a01d11d0 0x7fb1a00700f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb190004650 tx=0x7fb190004a50 comp rx=0 tx=0).stop 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a01d11d0 msgr2=0x7fb1a00700f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb197fff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a01d11d0 0x7fb1a00700f0 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb190004650 tx=0x7fb190004a50 comp rx=0 tx=0).stop 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 52, 2024-09-11T05:20:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 520932, 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:20:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:20:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:20:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:20:17.322 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 10 librados: watch_flush enter 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 10 librados: watch_flush exit 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 20 client.64614.objecter shutdown clearing up homeless session... 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 10 client.64614.objecter successfully canceled tick 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb18403ced0 msgr2=0x7fb18403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 1 --2- 172.21.15.138:0/2168231951 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb18403ced0 0x7fb18403f390 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7fb1a006e190 tx=0x7fb1980a9070 comp rx=0 tx=0).stop 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 10 monclient: shutdown 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0134850 msgr2=0x7fb1a01d0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 1 --2- 172.21.15.138:0/2168231951 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb1a0134850 0x7fb1a01d0750 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb18800c3c0 tx=0x7fb18800c880 comp rx=0 tx=0).stop 2024-09-11T05:20:17.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb19ffff640 1 -- 172.21.15.138:0/2168231951 reap_dead start 2024-09-11T05:20:17.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb19ffff640 1 -- 172.21.15.138:0/2168231951 reap_dead start 2024-09-11T05:20:17.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 shutdown_connections 2024-09-11T05:20:17.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 >> 172.21.15.138:0/2168231951 conn(0x7fb1a0063640 msgr2=0x7fb1a0133af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:17.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.302+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 shutdown_connections 2024-09-11T05:20:17.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.306+0000 7fb1a5eff640 1 -- 172.21.15.138:0/2168231951 wait complete. 2024-09-11T05:20:17.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.306+0000 7fb1a5eff640 1 librados: shutdown 2024-09-11T05:20:17.325 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:20:17.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:20:17.326 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:20:17.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.430+0000 7f1ebfdaa640 10 monclient: get_monmap_and_config 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.430+0000 7f1ebfdaa640 10 monclient: build_initial_monmap 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.430+0000 7f1ebfdaa640 10 monclient: monmap: 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:17.437361+0000 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:17.437361+0000 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.432 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:17.433 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:17.433 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:17.433 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:17.433 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:17.433 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 1 Processor -- start 2024-09-11T05:20:17.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 1 -- start start 2024-09-11T05:20:17.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient: init 2024-09-11T05:20:17.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:17.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-11T05:20:17.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1eb8070010 0x7f1eb8135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb8070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:17.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1eb809e6e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb809eb10 0x7f1eb81389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb809eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eb80a8990 con 0x7f1eb809e6e0 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb80703f0 con 0x7f1eb8070010 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb809cbf0 con 0x7f1eb809eb10 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebfdaa640 10 monclient(hunting): authenticate will time out at 1700.873504s 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb809eb10 0x7f1eb81389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb809eb10 0x7f1eb81389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:40822/0 (socket says 172.21.15.138:40822) 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 1 -- 172.21.15.138:0/3041846505 learned_addr learned my addr 172.21.15.138:0/3041846505 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:17.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient(hunting): get_auth_request con 0x7f1eb809eb10 auth_method 0 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebdb46640 1 --2- 172.21.15.138:0/3041846505 >> v2:172.21.15.190:3300/0 conn(0x7f1eb8070010 0x7f1eb8135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebdb46640 10 monclient(hunting): get_auth_request con 0x7f1eb8070010 auth_method 0 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebdb46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebdb46640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebdb46640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient(hunting): handle_auth_done global_id 84296 payload 274 2024-09-11T05:20:17.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 1 -- 172.21.15.138:0/3041846505 >> v2:172.21.15.190:3300/0 conn(0x7f1eb8070010 msgr2=0x7f1eb8135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:17.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 1 --2- 172.21.15.138:0/3041846505 >> v2:172.21.15.190:3300/0 conn(0x7f1eb8070010 0x7f1eb8135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 1 -- 172.21.15.138:0/3041846505 >> v1:172.21.15.190:6789/0 conn(0x7f1eb809e6e0 legacy=0x7f1eb81382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:17.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient: _finish_hunting 0 2024-09-11T05:20:17.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 1 monclient: found mon.noname-b 2024-09-11T05:20:17.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:17.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:20:17.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.434+0000 7f1ebe347640 1 -- 172.21.15.138:0/3041846505 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb809ced0 con 0x7f1eb809eb10 2024-09-11T05:20:17.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebe347640 1 -- 172.21.15.138:0/3041846505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb809eb10 msgr2=0x7f1eb81389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebe347640 1 --2- 172.21.15.138:0/3041846505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb809eb10 0x7f1eb81389d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009e50 tx=0x7f1eb001c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a4303a8423f5955 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 1 -- 172.21.15.138:0/3041846505 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb00a5720 con 0x7f1eb809eb10 2024-09-11T05:20:17.441 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: dump: 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: _finish_auth 0 2024-09-11T05:20:17.442 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: _check_auth_tickets 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 1 -- 172.21.15.138:0/3041846505 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb00a58c0 con 0x7f1eb809eb10 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 5 monclient: authenticate success, global_id 84296 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 4 set_mon_vals no callback set 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 1 -- 172.21.15.138:0/3041846505 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb00a5c90 con 0x7f1eb809eb10 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebcb44640 10 monclient: dump: 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.443 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 10 monclient: shutdown 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/3041846505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb809eb10 msgr2=0x7f1eb81389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 --2- 172.21.15.138:0/3041846505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb809eb10 0x7f1eb81389d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009e50 tx=0x7f1eb001c8c0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/3041846505 shutdown_connections 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 --2- 172.21.15.138:0/3041846505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb809eb10 0x7f1eb81389d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 --2- 172.21.15.138:0/3041846505 >> v2:172.21.15.190:3300/0 conn(0x7f1eb8070010 0x7f1eb8135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/3041846505 >> 172.21.15.138:0/3041846505 conn(0x7f1eb80ab830 msgr2=0x7f1eb80a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/3041846505 shutdown_connections 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/3041846505 wait complete. 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 10 monclient: build_initial_monmap 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 10 monclient: monmap: 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:17.445 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 librados: starting msgr at 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 librados: starting objecter 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.438+0000 7f1ebfdaa640 1 Processor -- start 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 -- start start 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 librados: setting wanted keys 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 librados: calling monclient init 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient: init 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb8070010 0x7f1eb812ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb8070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1eb809e6e0 0x7f1eb812f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb809e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:17.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb809eb10 0x7f1eb81d0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb809eb10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb8142e80 con 0x7f1eb809e6e0 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb8139120 con 0x7f1eb809eb10 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb8143000 con 0x7f1eb8070010 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): authenticate will time out at 1700.880466s 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb8070010 0x7f1eb812ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1eb809e6e0 0x7f1eb812f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb8070010 0x7f1eb812ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:40836/0 (socket says 172.21.15.138:40836) 2024-09-11T05:20:17.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 -- 172.21.15.138:0/2828261142 learned_addr learned my addr 172.21.15.138:0/2828261142 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient(hunting): get_auth_request con 0x7f1eb8070010 auth_method 0 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 10 monclient(hunting): get_auth_request con 0x7f1eb809e6e0 auth_method 0 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebe347640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb809eb10 0x7f1eb81d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient(hunting): handle_auth_done global_id 84299 payload 824 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb809eb10 msgr2=0x7f1eb81d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb809eb10 0x7f1eb81d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1eb809e6e0 msgr2=0x7f1eb812f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1eb809e6e0 0x7f1eb812f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient: _finish_hunting 0 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 monclient: found mon.noname-c 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 -- 172.21.15.138:0/2828261142 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb809fbd0 con 0x7f1eb8070010 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebe347640 10 monclient: get_auth_request con 0x7f1eb809eb10 auth_method 0 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebd345640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1eb809e6e0 0x7f1eb812f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:17.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebe347640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1eb809eb10 0x7f1eb81d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:20:17.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb8070010 msgr2=0x7f1eb812ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.453 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebdb46640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb8070010 0x7f1eb812ebb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400d800 tx=0x7f1eb400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 client.?.objecter ms_handle_connect 0x7f1eb8070010 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb400b8d0 con 0x7f1eb8070010 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: dump: 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.454 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: _finish_auth 0 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb4019070 con 0x7f1eb8070010 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 5 monclient: authenticate success, global_id 84299 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient: _renew_subs 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb8138b30 con 0x7f1eb8070010 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb400bc70 con 0x7f1eb8070010 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eae7fc640 10 monclient: dump: 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1eaffff640 4 set_mon_vals no callback set 2024-09-11T05:20:17.456 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 client.84299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient: _renew_subs 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb80a0780 con 0x7f1eb8070010 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 librados: init done 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb80a09c0 0x7f1eb81d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.442+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb8135450 con 0x7f1eb80a09c0 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb80a09c0 0x7f1eb81d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 10 monclient: get_auth_request con 0x7f1eb80a09c0 auth_method 0 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1eb4013070 con 0x7f1eb8070010 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 10 monclient(hunting): handle_auth_done global_id 84302 payload 824 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 1 -- 172.21.15.138:0/2828261142 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1eb80a0da0 con 0x7f1eb80a09c0 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 --2- 172.21.15.138:0/2828261142 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1e9c03cef0 0x7f1e9c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(518..518 src has 269..518) ==== 5464+0+0 (secure 0 0 0) 0x7f1eb404fb60 con 0x7f1eb8070010 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 client.84299.objecter ms_dispatch 0x7f1eb8000ba0 osd_map(518..518 src has 269..518) 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 3 client.84299.objecter handle_osd_map got epochs [518,518] > 0 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 3 client.84299.objecter handle_osd_map decoding full epoch 518 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 20 client.84299.objecter dump_active .. 0 homeless 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebe347640 1 --2- 172.21.15.138:0/2828261142 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1e9c03cef0 0x7f1e9c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebe347640 10 monclient: get_auth_request con 0x7f1e9c03cef0 auth_method 0 2024-09-11T05:20:17.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 client.84299.objecter ms_handle_connect 0x7f1eb80a09c0 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 client.84299.objecter resend_mon_ops 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb80a09c0 msgr2=0x7f1eb81d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebd345640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb80a09c0 0x7f1eb81d07d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8007fd0 tx=0x7f1ea800ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8018020 con 0x7f1eb80a09c0 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 monclient: dump: 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.459 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1ea800dd60 con 0x7f1eb80a09c0 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb80a09c0 msgr2=0x7f1eb81d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb80a09c0 0x7f1eb81d07d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8007fd0 tx=0x7f1ea800ff70 comp rx=0 tx=0).stop 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb80a09c0 msgr2=0x7f1eb81d07d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb80a09c0 0x7f1eb81d07d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1eae7fc640 10 client.84299.objecter ms_handle_connect 0x7f1e9c03cef0 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.446+0000 7f1ebe347640 1 --2- 172.21.15.138:0/2828261142 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1e9c03cef0 0x7f1e9c03f3b0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f1eb806efb0 tx=0x7f1eb00a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebfdaa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebfdaa640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb81d0d10 0x7f1eb80a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebfdaa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb81d10f0 con 0x7f1eb81d0d10 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb81d0d10 0x7f1eb80a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 10 monclient: get_auth_request con 0x7f1eb81d0d10 auth_method 0 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 10 monclient(hunting): handle_auth_done global_id 84305 payload 824 2024-09-11T05:20:17.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 1 -- 172.21.15.138:0/2828261142 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1eb8069fa0 con 0x7f1eb81d0d10 2024-09-11T05:20:17.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb81d0d10 msgr2=0x7f1eb80a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 10 client.84299.objecter ms_handle_connect 0x7f1eb81d0d10 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1ebd345640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb81d0d10 0x7f1eb80a2540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8019000 tx=0x7f1ea8003700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 10 client.84299.objecter resend_mon_ops 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8018030 con 0x7f1eb81d0d10 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 10 monclient: dump: 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.463 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f1ea801d420 con 0x7f1eb81d0d10 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb81d0d10 msgr2=0x7f1eb80a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb81d0d10 0x7f1eb80a2540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8019000 tx=0x7f1ea8003700 comp rx=0 tx=0).stop 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb81d0d10 msgr2=0x7f1eb80a2540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.454+0000 7f1eae7fc640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb81d0d10 0x7f1eb80a2540 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8019000 tx=0x7f1ea8003700 comp rx=0 tx=0).stop 2024-09-11T05:20:17.464 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 52, 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 261224, 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:17.465 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.466 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:20:17.468 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:17.469 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:17.470 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:17.471 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.472 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:17.473 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:17.475 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:20:17.476 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 10 librados: watch_flush enter 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 10 librados: watch_flush exit 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 20 client.84299.objecter shutdown clearing up homeless session... 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 10 client.84299.objecter successfully canceled tick 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1e9c03cef0 msgr2=0x7f1e9c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 --2- 172.21.15.138:0/2828261142 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1e9c03cef0 0x7f1e9c03f3b0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f1eb806efb0 tx=0x7f1eb00a7070 comp rx=0 tx=0).stop 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 10 monclient: shutdown 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:17.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb8070010 msgr2=0x7f1eb812ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 --2- 172.21.15.138:0/2828261142 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1eb8070010 0x7f1eb812ebb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400d800 tx=0x7f1eb400dcc0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebe347640 1 -- 172.21.15.138:0/2828261142 reap_dead start 2024-09-11T05:20:17.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebe347640 1 -- 172.21.15.138:0/2828261142 reap_dead start 2024-09-11T05:20:17.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 shutdown_connections 2024-09-11T05:20:17.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 >> 172.21.15.138:0/2828261142 conn(0x7f1eb80ab830 msgr2=0x7f1eb8131b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:17.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 shutdown_connections 2024-09-11T05:20:17.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 -- 172.21.15.138:0/2828261142 wait complete. 2024-09-11T05:20:17.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.458+0000 7f1ebfdaa640 1 librados: shutdown 2024-09-11T05:20:17.479 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:20:17.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:20:17.480 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:20:17.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient: get_monmap_and_config 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient: build_initial_monmap 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient: monmap: 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:17.590403+0000 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:17.590403+0000 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:17.585 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:17.586 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:17.586 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 1 Processor -- start 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 1 -- start start 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient: init 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd35c0a32c0 0x7fd35c09e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient(hunting): picked mon.noname-a con 0x7fd35c0a32c0 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd35c09e960 0x7fd35c09ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient(hunting): picked mon.noname-c con 0x7fd35c09e960 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd35c139710 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd35c0a1920 con 0x7fd35c139710 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fd35c0744a0 con 0x7fd35c0a32c0 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd35c0a36a0 con 0x7fd35c09e960 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd36260b640 10 monclient(hunting): authenticate will time out at 1701.026979s 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd35c0a32c0 0x7fd35c09e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd35c0a32c0 0x7fd35c09e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34290/0 (socket says 172.21.15.138:34290) 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 1 -- 172.21.15.138:0/4219522932 learned_addr learned my addr 172.21.15.138:0/4219522932 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient(hunting): get_auth_request con 0x7fd35c0a32c0 auth_method 0 2024-09-11T05:20:17.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35b7fe640 1 -- 172.21.15.138:0/4219522932 >> v2:172.21.15.190:3300/0 conn(0x7fd35c09e960 msgr2=0x7fd35c09ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35b7fe640 1 -- 172.21.15.138:0/4219522932 >> v2:172.21.15.190:3300/0 conn(0x7fd35c09e960 msgr2=0x7fd35c09ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35b7fe640 1 --2- 172.21.15.138:0/4219522932 >> v2:172.21.15.190:3300/0 conn(0x7fd35c09e960 0x7fd35c09ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35b7fe640 1 --2- 172.21.15.138:0/4219522932 >> v2:172.21.15.190:3300/0 conn(0x7fd35c09e960 0x7fd35c09ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient(hunting): handle_auth_done global_id 64626 payload 274 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 1 -- 172.21.15.138:0/4219522932 >> v2:172.21.15.190:3300/0 conn(0x7fd35c09e960 msgr2=0x7fd35c09ed70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 1 --2- 172.21.15.138:0/4219522932 >> v2:172.21.15.190:3300/0 conn(0x7fd35c09e960 0x7fd35c09ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 1 -- 172.21.15.138:0/4219522932 >> v1:172.21.15.190:6789/0 conn(0x7fd35c139710 legacy=0x7fd35c13bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:17.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:20:17.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 1 monclient: found mon.noname-a 2024-09-11T05:20:17.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:17.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:17.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.586+0000 7fd35bfff640 1 -- 172.21.15.138:0/4219522932 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35c0644b0 con 0x7fd35c0a32c0 2024-09-11T05:20:17.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35bfff640 1 -- 172.21.15.138:0/4219522932 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 msgr2=0x7fd35c09e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35bfff640 1 --2- 172.21.15.138:0/4219522932 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 0x7fd35c09e420 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd35c06abd0 tx=0x7fd34c01e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb8d3a4ffbc67f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 1 -- 172.21.15.138:0/4219522932 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0b4070 con 0x7fd35c0a32c0 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: dump: 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.595 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: _finish_auth 0 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: _check_auth_tickets 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 1 -- 172.21.15.138:0/4219522932 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd34c0a7960 con 0x7fd35c0a32c0 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 1 -- 172.21.15.138:0/4219522932 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0a7d30 con 0x7fd35c0a32c0 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd36260b640 5 monclient: authenticate success, global_id 64626 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd36260b640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd36260b640 4 set_mon_vals no callback set 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:17.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.590+0000 7fd35affd640 10 monclient: dump: 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient: shutdown 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:17.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- 172.21.15.138:0/4219522932 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 msgr2=0x7fd35c09e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 --2- 172.21.15.138:0/4219522932 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 0x7fd35c09e420 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd35c06abd0 tx=0x7fd34c01e8f0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- 172.21.15.138:0/4219522932 shutdown_connections 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 --2- 172.21.15.138:0/4219522932 >> v2:172.21.15.190:3300/0 conn(0x7fd35c09e960 0x7fd35c09ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 --2- 172.21.15.138:0/4219522932 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 0x7fd35c09e420 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- 172.21.15.138:0/4219522932 >> 172.21.15.138:0/4219522932 conn(0x7fd35c063640 msgr2=0x7fd35c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- 172.21.15.138:0/4219522932 shutdown_connections 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- 172.21.15.138:0/4219522932 wait complete. 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient: build_initial_monmap 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient: monmap: 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:17.598 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 librados: starting msgr at 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 librados: starting objecter 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 Processor -- start 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- start start 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 librados: setting wanted keys 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 librados: calling monclient init 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient: init 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:20:17.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c09e960 0x7fd35c1217c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient(hunting): picked mon.noname-b con 0x7fd35c09e960 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 0x7fd35c1c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient(hunting): picked mon.noname-a con 0x7fd35c0a32c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd35c139710 0x7fd35c138610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient(hunting): picked mon.noname-c con 0x7fd35c139710 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c0a29c0 con 0x7fd35c0a32c0 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c09f590 con 0x7fd35c09e960 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd35c0a2b40 con 0x7fd35c139710 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:17.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd36260b640 10 monclient(hunting): authenticate will time out at 1701.033854s 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 0x7fd35c1c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd35c139710 0x7fd35c138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 0x7fd35c1c3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34304/0 (socket says 172.21.15.138:34304) 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 1 -- 172.21.15.138:0/2740995125 learned_addr learned my addr 172.21.15.138:0/2740995125 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 10 monclient(hunting): get_auth_request con 0x7fd35c0a32c0 auth_method 0 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient(hunting): get_auth_request con 0x7fd35c139710 auth_method 0 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c09e960 0x7fd35c1217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35bfff640 10 monclient(hunting): get_auth_request con 0x7fd35c09e960 auth_method 0 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient(hunting): handle_auth_done global_id 84308 payload 824 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c09e960 msgr2=0x7fd35c1217c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c09e960 0x7fd35c1217c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 msgr2=0x7fd35c1c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:17.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 0x7fd35c1c3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:17.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient: _finish_hunting 0 2024-09-11T05:20:17.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 1 monclient: found mon.noname-c 2024-09-11T05:20:17.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:17.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:17.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd360ba8640 1 -- 172.21.15.138:0/2740995125 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35c09d500 con 0x7fd35c139710 2024-09-11T05:20:17.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.594+0000 7fd35b7fe640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd35c0a32c0 0x7fd35c1c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:17.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd360ba8640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd35c139710 msgr2=0x7fd35c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd360ba8640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd35c139710 0x7fd35c138610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd35400c780 tx=0x7fd35400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 client.?.objecter ms_handle_connect 0x7fd35c139710 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354016020 con 0x7fd35c139710 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: dump: 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.606 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: _finish_auth 0 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd35400ac60 con 0x7fd35c139710 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354012660 con 0x7fd35c139710 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 5 monclient: authenticate success, global_id 84308 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 10 monclient: _renew_subs 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35c1392d0 con 0x7fd35c139710 2024-09-11T05:20:17.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35a7fc640 4 set_mon_vals no callback set 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 monclient: dump: 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 10 client.84308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:17.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 10 monclient: _renew_subs 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd35c138b50 con 0x7fd35c139710 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 1 librados: init done 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c138db0 con 0x7fd35c11f960 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fd3540128c0 con 0x7fd35c139710 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 10 monclient: get_auth_request con 0x7fd35c11f960 auth_method 0 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 1 --2- 172.21.15.138:0/2740995125 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd34803cf00 0x7fd34803f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:17.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(518..518 src has 269..518) ==== 5464+0+0 (secure 0 0 0) 0x7fd35404fc90 con 0x7fd35c139710 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 10 client.84308.objecter ms_dispatch 0x7fd35c000ba0 osd_map(518..518 src has 269..518) 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 3 client.84308.objecter handle_osd_map got epochs [518,518] > 0 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 3 client.84308.objecter handle_osd_map decoding full epoch 518 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd358ff9640 20 client.84308.objecter dump_active .. 0 homeless 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35b7fe640 1 --2- 172.21.15.138:0/2740995125 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd34803cf00 0x7fd34803f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.598+0000 7fd35b7fe640 10 monclient: get_auth_request con 0x7fd34803cf00 auth_method 0 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.602+0000 7fd35b7fe640 1 --2- 172.21.15.138:0/2740995125 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd34803cf00 0x7fd34803f3c0 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7fd35c09e2a0 tx=0x7fd3500091c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:17.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.602+0000 7fd358ff9640 10 client.84308.objecter ms_handle_connect 0x7fd34803cf00 2024-09-11T05:20:17.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:17.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 10 monclient: get_auth_request con 0x7fd35c11f960 auth_method 0 2024-09-11T05:20:17.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:17.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:17.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:17.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:20:17.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:20:17.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:17.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:17.802+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 10 monclient: get_auth_request con 0x7fd35c11f960 auth_method 0 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:18.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:18.206+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:20:19.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:19.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 10 monclient: get_auth_request con 0x7fd35c11f960 auth_method 0 2024-09-11T05:20:19.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:19.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:19.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:19.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:20:19.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:20:19.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:19.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:19.010+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:20:20.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.594+0000 7fd3437fe640 10 monclient: tick 2024-09-11T05:20:20.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.594+0000 7fd3437fe640 10 monclient: _check_auth_tickets 2024-09-11T05:20:20.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.594+0000 7fd3437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:20.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.594+0000 7fd3437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:20.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:20.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 10 monclient: get_auth_request con 0x7fd35c11f960 auth_method 0 2024-09-11T05:20:20.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:20.613 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:20.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:20.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:20.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:20.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:20.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 10 monclient(hunting): handle_auth_done global_id 104098 payload 824 2024-09-11T05:20:20.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.614+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd35c0a2f90 con 0x7fd35c11f960 2024-09-11T05:20:20.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 10 client.84308.objecter ms_handle_connect 0x7fd35c11f960 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 10 client.84308.objecter resend_mon_ops 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd35c09f7e0 tx=0x7fd34c01b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0c8070 con 0x7fd35c11f960 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 10 monclient: dump: 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:20.617 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.618 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.618 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:20.618 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:20.618 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:20.618 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:20.618 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:20.618 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:20.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd34c00b730 con 0x7fd35c11f960 2024-09-11T05:20:20.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:20.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:20.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:20.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd35c09f7e0 tx=0x7fd34c01b690 comp rx=0 tx=0).stop 2024-09-11T05:20:20.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 msgr2=0x7fd35c02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:20.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.618+0000 7fd358ff9640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11f960 0x7fd35c02d450 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd35c09f7e0 tx=0x7fd34c01b690 comp rx=0 tx=0).stop 2024-09-11T05:20:20.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd36260b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:20:20.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd36260b640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11fd40 0x7fd35c02d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:20.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd36260b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:20.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c1227d0 con 0x7fd35c11fd40 2024-09-11T05:20:20.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11fd40 0x7fd35c02d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 10 monclient: get_auth_request con 0x7fd35c11fd40 auth_method 0 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 10 monclient(hunting): handle_auth_done global_id 104101 payload 824 2024-09-11T05:20:20.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.626+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd35c072a00 con 0x7fd35c11fd40 2024-09-11T05:20:20.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 10 client.84308.objecter ms_handle_connect 0x7fd35c11fd40 2024-09-11T05:20:20.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd35bfff640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11fd40 msgr2=0x7fd35c02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:20.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 10 client.84308.objecter resend_mon_ops 2024-09-11T05:20:20.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd35bfff640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11fd40 0x7fd35c02d990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd34c01cfa0 tx=0x7fd34c00b450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0c8040 con 0x7fd35c11fd40 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 10 monclient: dump: 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:20.630 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:20.631 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:20.631 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:20.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fd34c0ca2d0 con 0x7fd35c11fd40 2024-09-11T05:20:20.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:20:20.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:20.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11fd40 msgr2=0x7fd35c02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:20.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11fd40 0x7fd35c02d990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd34c01cfa0 tx=0x7fd34c00b450 comp rx=0 tx=0).stop 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11fd40 msgr2=0x7fd35c02d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd358ff9640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd35c11fd40 0x7fd35c02d990 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd34c01cfa0 tx=0x7fd34c00b450 comp rx=0 tx=0).stop 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 52, 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 10795, 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:20:20.632 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 4, 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:20.633 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:20.634 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:20:20.635 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:20:20.636 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:20.637 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.638 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:20.639 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:20.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:20.641 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:20.642 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:20.643 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:20.644 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:20.644 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:20.644 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:20.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:20.644 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:20:20.644 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:20.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 10 librados: watch_flush enter 2024-09-11T05:20:20.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 10 librados: watch_flush exit 2024-09-11T05:20:20.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 20 client.84308.objecter shutdown clearing up homeless session... 2024-09-11T05:20:20.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 10 client.84308.objecter successfully canceled tick 2024-09-11T05:20:20.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd34803cf00 msgr2=0x7fd34803f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:20.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 1 --2- 172.21.15.138:0/2740995125 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd34803cf00 0x7fd34803f3c0 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7fd35c09e2a0 tx=0x7fd3500091c0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 10 monclient: shutdown 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd360ba8640 1 -- 172.21.15.138:0/2740995125 reap_dead start 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd35c139710 msgr2=0x7fd35c138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd35c139710 0x7fd35c138610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd35400c780 tx=0x7fd35400cc40 comp rx=0 tx=0).stop 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 shutdown_connections 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 1 --2- 172.21.15.138:0/2740995125 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd35c139710 0x7fd35c138610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 >> 172.21.15.138:0/2740995125 conn(0x7fd35c063640 msgr2=0x7fd35c0a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.630+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 shutdown_connections 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.634+0000 7fd36260b640 1 -- 172.21.15.138:0/2740995125 wait complete. 2024-09-11T05:20:20.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.634+0000 7fd36260b640 1 librados: shutdown 2024-09-11T05:20:20.656 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:20:20.656 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:20:20.657 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:20:20.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.778+0000 7f48e2b2c640 10 monclient: get_monmap_and_config 2024-09-11T05:20:20.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.778+0000 7f48e2b2c640 10 monclient: build_initial_monmap 2024-09-11T05:20:20.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.778+0000 7f48e2b2c640 10 monclient: monmap: 2024-09-11T05:20:20.779 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:20.779 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:20.779 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:20.784865+0000 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:20.784865+0000 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:20.780 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 1 Processor -- start 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 1 -- start start 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient: init 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc09e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient(hunting): picked mon.noname-b con 0x7f48dc06e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f48dc09e960 0x7f48dc09ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient(hunting): picked mon.noname-c con 0x7f48dc09e960 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f48dc1356f0 0x7f48dc135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient(hunting): picked mon.noname-a con 0x7f48dc1356f0 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:20.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f48dc09f280 con 0x7f48dc1356f0 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f48dc06e4c0 con 0x7f48dc09e960 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc09f820 con 0x7f48dc06e0e0 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e2b2c640 10 monclient(hunting): authenticate will time out at 1704.221042s 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc09e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f48dc1356f0 0x7f48dc135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f48dc1356f0 0x7f48dc135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34312/0 (socket says 172.21.15.138:34312) 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 -- 172.21.15.138:0/2443635493 learned_addr learned my addr 172.21.15.138:0/2443635493 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient(hunting): get_auth_request con 0x7f48dc1356f0 auth_method 0 2024-09-11T05:20:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc09e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:47968/0 (socket says 172.21.15.138:47968) 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 10 monclient(hunting): get_auth_request con 0x7f48dc06e0e0 auth_method 0 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:20.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48dbfff640 1 --2- 172.21.15.138:0/2443635493 >> v2:172.21.15.190:3300/0 conn(0x7f48dc09e960 0x7f48dc09ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48dbfff640 10 monclient(hunting): get_auth_request con 0x7f48dc09e960 auth_method 0 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient(hunting): handle_auth_done global_id 64632 payload 274 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 -- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 msgr2=0x7f48dc09e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 --2- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc09e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 -- 172.21.15.138:0/2443635493 >> v2:172.21.15.190:3300/0 conn(0x7f48dc09e960 msgr2=0x7f48dc09ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 --2- 172.21.15.138:0/2443635493 >> v2:172.21.15.190:3300/0 conn(0x7f48dc09e960 0x7f48dc09ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient: _finish_hunting 0 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 monclient: found mon.noname-a 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:20.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:20.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e10c9640 1 -- 172.21.15.138:0/2443635493 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc09f540 con 0x7f48dc1356f0 2024-09-11T05:20:20.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.782+0000 7f48e08c8640 1 --2- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc09e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e10c9640 1 -- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc1356f0 msgr2=0x7f48dc135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e10c9640 1 --2- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc1356f0 0x7f48dc135b40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f48dc069420 tx=0x7f48d401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339c9da78ff30128 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 1 -- 172.21.15.138:0/2443635493 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d40a56b0 con 0x7f48dc1356f0 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: dump: 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:20.790 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: _finish_auth 0 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 1 -- 172.21.15.138:0/2443635493 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d40a5850 con 0x7f48dc1356f0 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 1 -- 172.21.15.138:0/2443635493 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d40a5c20 con 0x7f48dc1356f0 2024-09-11T05:20:20.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 5 monclient: authenticate success, global_id 64632 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 4 set_mon_vals no callback set 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48db7fe640 10 monclient: dump: 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:20.792 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 10 monclient: shutdown 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 -- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc1356f0 msgr2=0x7f48dc135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc1356f0 0x7f48dc135b40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f48dc069420 tx=0x7f48d401c910 comp rx=0 tx=0).stop 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 -- 172.21.15.138:0/2443635493 shutdown_connections 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc1356f0 0x7f48dc135b40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/2443635493 >> v2:172.21.15.190:3300/0 conn(0x7f48dc09e960 0x7f48dc09ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/2443635493 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc09e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 -- 172.21.15.138:0/2443635493 >> 172.21.15.138:0/2443635493 conn(0x7f48dc063640 msgr2=0x7f48dc063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 -- 172.21.15.138:0/2443635493 shutdown_connections 2024-09-11T05:20:20.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 -- 172.21.15.138:0/2443635493 wait complete. 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 10 monclient: build_initial_monmap 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 10 monclient: monmap: 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 librados: starting msgr at 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 librados: starting objecter 2024-09-11T05:20:20.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 Processor -- start 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 -- start start 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 librados: setting wanted keys 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 1 librados: calling monclient init 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.786+0000 7f48e2b2c640 10 monclient: init 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc11e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient(hunting): picked mon.noname-c con 0x7f48dc06e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc09e960 0x7f48dc134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient(hunting): picked mon.noname-a con 0x7f48dc09e960 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48dc1356f0 0x7f48dc11d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient(hunting): picked mon.noname-b con 0x7f48dc1356f0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc0a1430 con 0x7f48dc09e960 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:20.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc0a12b0 con 0x7f48dc1356f0 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc0a29c0 con 0x7f48dc06e0e0 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient(hunting): authenticate will time out at 1704.228411s 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc11e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc09e960 0x7f48dc134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc11e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:47982/0 (socket says 172.21.15.138:47982) 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 -- 172.21.15.138:0/3658937019 learned_addr learned my addr 172.21.15.138:0/3658937019 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient(hunting): get_auth_request con 0x7f48dc06e0e0 auth_method 0 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc09e960 0x7f48dc134400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34322/0 (socket says 172.21.15.138:34322) 2024-09-11T05:20:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 10 monclient(hunting): get_auth_request con 0x7f48dc09e960 auth_method 0 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient(hunting): handle_auth_done global_id 84314 payload 824 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 -- 172.21.15.138:0/3658937019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48dc1356f0 msgr2=0x7f48dc11d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48dc1356f0 0x7f48dc11d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 -- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc09e960 msgr2=0x7f48dc134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc09e960 0x7f48dc134400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:20.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient: _finish_hunting 0 2024-09-11T05:20:20.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 monclient: found mon.noname-c 2024-09-11T05:20:20.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:20.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:20.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 -- 172.21.15.138:0/3658937019 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc063c90 con 0x7f48dc06e0e0 2024-09-11T05:20:20.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48dbfff640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc09e960 0x7f48dc134400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:20.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 -- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 msgr2=0x7f48dc11e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:20.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e08c8640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc11e5c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f48d0010e60 tx=0x7f48d000e730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:20.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 client.?.objecter ms_handle_connect 0x7f48dc06e0e0 2024-09-11T05:20:20.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 1 -- 172.21.15.138:0/3658937019 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d001e070 con 0x7f48dc06e0e0 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: dump: 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:20.801 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: _finish_auth 0 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: _check_auth_tickets 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 1 -- 172.21.15.138:0/3658937019 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d0011a70 con 0x7f48dc06e0e0 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 5 monclient: authenticate success, global_id 84314 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 1 -- 172.21.15.138:0/3658937019 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d000ee80 con 0x7f48dc06e0e0 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient: _renew_subs 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48dc11fdc0 con 0x7f48dc06e0e0 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48daffd640 4 set_mon_vals no callback set 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 client.84314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient: _renew_subs 2024-09-11T05:20:20.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48dc11fae0 con 0x7f48dc06e0e0 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48e2b2c640 1 librados: init done 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.790+0000 7f48d97fa640 10 monclient: dump: 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:20.803 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48e2b2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48e2b2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48e2b2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48dc0a2e20 con 0x7f48dc06e0e0 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48d97fa640 1 -- 172.21.15.138:0/3658937019 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f48d000e040 con 0x7f48dc06e0e0 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48d97fa640 1 --2- 172.21.15.138:0/3658937019 >> v2:172.21.15.190:6800/1562782189 conn(0x7f48c003cf20 0x7f48c003f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48d97fa640 1 -- 172.21.15.138:0/3658937019 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(521..521 src has 269..521) ==== 5845+0+0 (secure 0 0 0) 0x7f48d0052aa0 con 0x7f48dc06e0e0 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48d97fa640 10 client.84314.objecter ms_dispatch 0x7f48dc000ba0 osd_map(521..521 src has 269..521) 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48d97fa640 3 client.84314.objecter handle_osd_map got epochs [521,521] > 0 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48d97fa640 3 client.84314.objecter handle_osd_map decoding full epoch 521 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48d97fa640 20 client.84314.objecter dump_active .. 0 homeless 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48dbfff640 1 --2- 172.21.15.138:0/3658937019 >> v2:172.21.15.190:6800/1562782189 conn(0x7f48c003cf20 0x7f48c003f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.794+0000 7f48dbfff640 10 monclient: get_auth_request con 0x7f48c003cf20 auth_method 0 2024-09-11T05:20:20.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.798+0000 7f48d97fa640 10 client.84314.objecter ms_handle_connect 0x7f48c003cf20 2024-09-11T05:20:20.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.798+0000 7f48dbfff640 1 --2- 172.21.15.138:0/3658937019 >> v2:172.21.15.190:6800/1562782189 conn(0x7f48c003cf20 0x7f48c003f3e0 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7f48dc11ffd0 tx=0x7f48c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:20.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.798+0000 7f48d97fa640 1 -- 172.21.15.138:0/3658937019 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f48d0028020 con 0x7f48dc06e0e0 2024-09-11T05:20:20.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.798+0000 7f48d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:20.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:20.798+0000 7f48d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:21.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.030+0000 7f48e2b2c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:20:21.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.030+0000 7f48e2b2c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:20:21.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.030+0000 7f48e2b2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:21.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.030+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f48dc11f190 con 0x7f48dc06e0e0 2024-09-11T05:20:21.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f48d97fa640 1 -- 172.21.15.138:0/3658937019 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f48dc063c90 con 0x7f48dc06e0e0 2024-09-11T05:20:21.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f48d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:20:21.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f48d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:21.034 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.034 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=9,logm=60} crc {auth=3678143871,config=692619927,health=730422057,kv=3908132897,logm=3331397036}) 2024-09-11T05:20:21.034 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=3352647547,config=692619927,health=814375510,kv=310889196,logm=1506630837}) 2024-09-11T05:20:21.034 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.034 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=9,logm=60} crc {auth=3678143871,config=692619927,health=730422057,kv=3908132897,logm=3331397036}) 2024-09-11T05:20:21.034 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.034+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=1,health=12,kv=9,logm=62} crc {auth=2684237692,config=1228700967,health=730422057,kv=2820790219,logm=700460623}) 2024-09-11T05:20:21.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 10 librados: watch_flush enter 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 10 librados: watch_flush exit 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 20 client.84314.objecter shutdown clearing up homeless session... 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 10 client.84314.objecter successfully canceled tick 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 >> v2:172.21.15.190:6800/1562782189 conn(0x7f48c003cf20 msgr2=0x7f48c003f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/3658937019 >> v2:172.21.15.190:6800/1562782189 conn(0x7f48c003cf20 0x7f48c003f3e0 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7f48dc11ffd0 tx=0x7f48c8008040 comp rx=0 tx=0).stop 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 10 monclient: shutdown 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 msgr2=0x7f48dc11e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc11e5c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f48d0010e60 tx=0x7f48d000e730 comp rx=0 tx=0).stop 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 shutdown_connections 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/3658937019 >> v2:172.21.15.190:6800/1562782189 conn(0x7f48c003cf20 0x7f48c003f3e0 unknown :-1 s=CLOSED pgs=3457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48dc1356f0 0x7f48dc11d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48dc09e960 0x7f48dc134400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:21.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 --2- 172.21.15.138:0/3658937019 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f48dc06e0e0 0x7f48dc11e5c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:21.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 >> 172.21.15.138:0/3658937019 conn(0x7f48dc063640 msgr2=0x7f48dc072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:21.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 shutdown_connections 2024-09-11T05:20:21.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 -- 172.21.15.138:0/3658937019 wait complete. 2024-09-11T05:20:21.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f48e2b2c640 1 librados: shutdown 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505902395}) 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3919024428}) 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505902395}) 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=190813077}) 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48130021}) 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616918007}) 2024-09-11T05:20:21.039 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48130021}) 2024-09-11T05:20:21.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.038+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479683714}) 2024-09-11T05:20:21.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169946944}) 2024-09-11T05:20:21.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4176602105}) 2024-09-11T05:20:21.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169946944}) 2024-09-11T05:20:21.041 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2186494962}) 2024-09-11T05:20:21.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839459918}) 2024-09-11T05:20:21.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347938178}) 2024-09-11T05:20:21.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839459918}) 2024-09-11T05:20:21.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=113674575}) 2024-09-11T05:20:21.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486363885}) 2024-09-11T05:20:21.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286078669}) 2024-09-11T05:20:21.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486363885}) 2024-09-11T05:20:21.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.042+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=394146705}) 2024-09-11T05:20:21.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472565939}) 2024-09-11T05:20:21.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3760547388}) 2024-09-11T05:20:21.045 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.045 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472565939}) 2024-09-11T05:20:21.045 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3050930566}) 2024-09-11T05:20:21.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179075528}) 2024-09-11T05:20:21.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2935478010}) 2024-09-11T05:20:21.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179075528}) 2024-09-11T05:20:21.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235821417}) 2024-09-11T05:20:21.047 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.048 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752598972}) 2024-09-11T05:20:21.048 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1877971550}) 2024-09-11T05:20:21.048 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.048 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752598972}) 2024-09-11T05:20:21.048 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.046+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3198290833}) 2024-09-11T05:20:21.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.050 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980431878}) 2024-09-11T05:20:21.050 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3304632373}) 2024-09-11T05:20:21.050 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.050 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980431878}) 2024-09-11T05:20:21.050 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519534534}) 2024-09-11T05:20:21.051 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341700793}) 2024-09-11T05:20:21.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386793499}) 2024-09-11T05:20:21.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341700793}) 2024-09-11T05:20:21.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.050+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2544896290}) 2024-09-11T05:20:21.053 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.054+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.053 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.054+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522216244}) 2024-09-11T05:20:21.053 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.054+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448239678}) 2024-09-11T05:20:21.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.054+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.054+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522216244}) 2024-09-11T05:20:21.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.054+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033027353}) 2024-09-11T05:20:21.060 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:20:21.177 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149667870}) 2024-09-11T05:20:21.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3852762026}) 2024-09-11T05:20:21.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149667870}) 2024-09-11T05:20:21.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.178+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1444196720}) 2024-09-11T05:20:21.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1317528263}) 2024-09-11T05:20:21.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614198617}) 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1317528263}) 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.182+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276663216}) 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=647931930}) 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3858287668}) 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=647931930}) 2024-09-11T05:20:21.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3049511792}) 2024-09-11T05:20:21.733 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.734+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.734+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400927829}) 2024-09-11T05:20:21.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.734+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=841196442}) 2024-09-11T05:20:21.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.734+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.734+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400927829}) 2024-09-11T05:20:21.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.734+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3371612325}) 2024-09-11T05:20:21.736 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.738+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.736 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.738+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459272983}) 2024-09-11T05:20:21.736 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.738+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=847662703}) 2024-09-11T05:20:21.736 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.738+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.736 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.738+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459272983}) 2024-09-11T05:20:21.736 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.738+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305632515}) 2024-09-11T05:20:21.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.990+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.990+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=664875881}) 2024-09-11T05:20:21.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.990+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2018965457}) 2024-09-11T05:20:21.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.990+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:21.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.990+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=664875881}) 2024-09-11T05:20:21.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:21.990+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084949447}) 2024-09-11T05:20:22.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.006+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.006+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1314709423}) 2024-09-11T05:20:22.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.006+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2328381432}) 2024-09-11T05:20:22.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.006+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.006+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1314709423}) 2024-09-11T05:20:22.008 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.006+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3560549406}) 2024-09-11T05:20:22.061 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:20:22.061 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:20:22.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient: get_monmap_and_config 2024-09-11T05:20:22.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient: build_initial_monmap 2024-09-11T05:20:22.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient: monmap: 2024-09-11T05:20:22.189 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:22.195040+0000 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:22.195040+0000 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:22.190 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.191 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.191 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075407146}) 2024-09-11T05:20:22.191 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2157993486}) 2024-09-11T05:20:22.191 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.191 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075407146}) 2024-09-11T05:20:22.192 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3438234676}) 2024-09-11T05:20:22.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 1 Processor -- start 2024-09-11T05:20:22.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 1 -- start start 2024-09-11T05:20:22.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient: init 2024-09-11T05:20:22.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:22.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-11T05:20:22.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f71700a5b40 0x7f71700a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient(hunting): picked mon.noname-b con 0x7f71700a5b40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f71700a7130 0x7f71700a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient(hunting): picked mon.noname-a con 0x7f71700a7130 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f717009d840 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717009d490 con 0x7f717009d840 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f717009e110 con 0x7f71700a7130 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f71700a69a0 con 0x7f71700a5b40 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f7176416640 10 monclient(hunting): authenticate will time out at 1705.631213s 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f71700a5b40 0x7f71700a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f71700a7130 0x7f71700a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f71700a5b40 0x7f71700a5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:47998/0 (socket says 172.21.15.138:47998) 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f71700a7130 0x7f71700a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34334/0 (socket says 172.21.15.138:34334) 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 1 -- 172.21.15.138:0/903284281 learned_addr learned my addr 172.21.15.138:0/903284281 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 10 monclient(hunting): get_auth_request con 0x7f71700a5b40 auth_method 0 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient(hunting): get_auth_request con 0x7f71700a7130 auth_method 0 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient(hunting): handle_auth_done global_id 64638 payload 274 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 1 -- 172.21.15.138:0/903284281 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f71700a5b40 msgr2=0x7f71700a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 1 --2- 172.21.15.138:0/903284281 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f71700a5b40 0x7f71700a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 1 -- 172.21.15.138:0/903284281 >> v1:172.21.15.190:6789/0 conn(0x7f717009d840 legacy=0x7f7170138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 1 monclient: found mon.noname-a 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.190+0000 7f716f7fe640 1 -- 172.21.15.138:0/903284281 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7170070580 con 0x7f71700a7130 2024-09-11T05:20:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716f7fe640 1 -- 172.21.15.138:0/903284281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a7130 msgr2=0x7f71700a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716f7fe640 1 --2- 172.21.15.138:0/903284281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a7130 0x7f71700a6460 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f71640097c0 tx=0x7f716401c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddef4d469bbb1faa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 1 -- 172.21.15.138:0/903284281 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640b0070 con 0x7f71700a7130 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: dump: 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: _finish_auth 0 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: _check_auth_tickets 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 1 -- 172.21.15.138:0/903284281 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71640ab600 con 0x7f71700a7130 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 1 -- 172.21.15.138:0/903284281 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640ab9d0 con 0x7f71700a7130 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f7176416640 5 monclient: authenticate success, global_id 64638 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f7176416640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f7176416640 4 set_mon_vals no callback set 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f716effd640 10 monclient: dump: 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.202 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.202 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454517480}) 2024-09-11T05:20:22.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154306578}) 2024-09-11T05:20:22.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454517480}) 2024-09-11T05:20:22.203 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.198+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1217954549}) 2024-09-11T05:20:22.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient: shutdown 2024-09-11T05:20:22.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:22.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- 172.21.15.138:0/903284281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a7130 msgr2=0x7f71700a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 --2- 172.21.15.138:0/903284281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a7130 0x7f71700a6460 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f71640097c0 tx=0x7f716401c300 comp rx=0 tx=0).stop 2024-09-11T05:20:22.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- 172.21.15.138:0/903284281 shutdown_connections 2024-09-11T05:20:22.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 --2- 172.21.15.138:0/903284281 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a7130 0x7f71700a6460 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 --2- 172.21.15.138:0/903284281 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f71700a5b40 0x7f71700a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.207 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- 172.21.15.138:0/903284281 >> 172.21.15.138:0/903284281 conn(0x7f71700aba10 msgr2=0x7f71700a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- 172.21.15.138:0/903284281 shutdown_connections 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- 172.21.15.138:0/903284281 wait complete. 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient: build_initial_monmap 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient: monmap: 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:22.208 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 librados: starting msgr at 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 librados: starting objecter 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 Processor -- start 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- start start 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 librados: setting wanted keys 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 librados: calling monclient init 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient: init 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f717009d840 0x7f71701d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient(hunting): picked mon.noname-c con 0x7f717009d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a5b40 0x7f7170137af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient(hunting): picked mon.noname-a con 0x7f71700a5b40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71700a7130 0x7f717013da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient(hunting): picked mon.noname-b con 0x7f71700a7130 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f7170142f00 con 0x7f71700a5b40 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7170142d80 con 0x7f71700a7130 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f7170143080 con 0x7f717009d840 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f7176416640 10 monclient(hunting): authenticate will time out at 1705.641781s 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f717009d840 0x7f71701d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f717009d840 0x7f71701d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48004/0 (socket says 172.21.15.138:48004) 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 -- 172.21.15.138:0/2959803772 learned_addr learned my addr 172.21.15.138:0/2959803772 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient(hunting): get_auth_request con 0x7f717009d840 auth_method 0 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f71749b3640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71700a7130 0x7f717013da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient(hunting): handle_auth_done global_id 84320 payload 824 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 -- 172.21.15.138:0/2959803772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71700a7130 msgr2=0x7f717013da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71700a7130 0x7f717013da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 -- 172.21.15.138:0/2959803772 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a5b40 msgr2=0x7f7170137af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a5b40 0x7f7170137af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:20:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 monclient: found mon.noname-c 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f716ffff640 1 -- 172.21.15.138:0/2959803772 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71701d13e0 con 0x7f717009d840 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f71749b3640 10 monclient: get_auth_request con 0x7f71700a7130 auth_method 0 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.202+0000 7f71749b3640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71700a7130 0x7f717013da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716ffff640 1 -- 172.21.15.138:0/2959803772 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f717009d840 msgr2=0x7f71701d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716ffff640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f717009d840 0x7f71701d04f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f716000e900 tx=0x7f716000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 1 -- 172.21.15.138:0/2959803772 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 client.?.objecter ms_handle_connect 0x7f717009d840 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 1 -- 172.21.15.138:0/2959803772 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160017070 con 0x7f717009d840 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: dump: 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: _finish_auth 0 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: _check_auth_tickets 2024-09-11T05:20:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 1 -- 172.21.15.138:0/2959803772 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f716000cc10 con 0x7f717009d840 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f7176416640 5 monclient: authenticate success, global_id 84320 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 1 -- 172.21.15.138:0/2959803772 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160013470 con 0x7f717009d840 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716cff9640 10 monclient: dump: 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.215 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.206+0000 7f716e7fc640 4 set_mon_vals no callback set 2024-09-11T05:20:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 10 monclient: _renew_subs 2024-09-11T05:20:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7170121be0 con 0x7f717009d840 2024-09-11T05:20:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 10 client.84320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 10 monclient: _renew_subs 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f71701221a0 con 0x7f717009d840 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 1 librados: init done 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7170138900 con 0x7f717009d840 2024-09-11T05:20:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.210+0000 7f716cff9640 1 -- 172.21.15.138:0/2959803772 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f7160024060 con 0x7f717009d840 2024-09-11T05:20:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716cff9640 1 --2- 172.21.15.138:0/2959803772 >> v2:172.21.15.190:6800/1562782189 conn(0x7f715c03cef0 0x7f715c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716cff9640 1 -- 172.21.15.138:0/2959803772 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7f71600503e0 con 0x7f717009d840 2024-09-11T05:20:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716cff9640 10 client.84320.objecter ms_dispatch 0x7f7170000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716cff9640 3 client.84320.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716cff9640 3 client.84320.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716cff9640 20 client.84320.objecter dump_active .. 0 homeless 2024-09-11T05:20:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716f7fe640 1 --2- 172.21.15.138:0/2959803772 >> v2:172.21.15.190:6800/1562782189 conn(0x7f715c03cef0 0x7f715c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716f7fe640 10 monclient: get_auth_request con 0x7f715c03cef0 auth_method 0 2024-09-11T05:20:22.219 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.219 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809345407}) 2024-09-11T05:20:22.219 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2878176156}) 2024-09-11T05:20:22.219 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.219 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809345407}) 2024-09-11T05:20:22.219 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2731311480}) 2024-09-11T05:20:22.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716f7fe640 1 --2- 172.21.15.138:0/2959803772 >> v2:172.21.15.190:6800/1562782189 conn(0x7f715c03cef0 0x7f715c03f3b0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7f71640ab000 tx=0x7f71640a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.219 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.214+0000 7f716cff9640 10 client.84320.objecter ms_handle_connect 0x7f715c03cef0 2024-09-11T05:20:22.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.218+0000 7f716cff9640 1 -- 172.21.15.138:0/2959803772 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f7160013db0 con 0x7f717009d840 2024-09-11T05:20:22.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.218+0000 7f716cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.220 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.218+0000 7f716cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:22.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.454+0000 7f7176416640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:20:22.454 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.454+0000 7f7176416640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:22.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.454+0000 7f7176416640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:22.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.454+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7170069fa0 con 0x7f717009d840 2024-09-11T05:20:22.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.454+0000 7f716cff9640 1 -- 172.21.15.138:0/2959803772 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f71600500d0 con 0x7f717009d840 2024-09-11T05:20:22.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.454+0000 7f716cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:22.455 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.454+0000 7f716cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:22.456 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:20:22.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 10 librados: watch_flush enter 2024-09-11T05:20:22.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 10 librados: watch_flush exit 2024-09-11T05:20:22.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 20 client.84320.objecter shutdown clearing up homeless session... 2024-09-11T05:20:22.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 10 client.84320.objecter successfully canceled tick 2024-09-11T05:20:22.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 >> v2:172.21.15.190:6800/1562782189 conn(0x7f715c03cef0 msgr2=0x7f715c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 --2- 172.21.15.138:0/2959803772 >> v2:172.21.15.190:6800/1562782189 conn(0x7f715c03cef0 0x7f715c03f3b0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7f71640ab000 tx=0x7f71640a6000 comp rx=0 tx=0).stop 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 10 monclient: shutdown 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f717009d840 msgr2=0x7f71701d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f717009d840 0x7f71701d04f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f716000e900 tx=0x7f716000edc0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 shutdown_connections 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 --2- 172.21.15.138:0/2959803772 >> v2:172.21.15.190:6800/1562782189 conn(0x7f715c03cef0 0x7f715c03f3b0 unknown :-1 s=CLOSED pgs=3458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71700a7130 0x7f717013da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f71700a5b40 0x7f7170137af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 --2- 172.21.15.138:0/2959803772 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f717009d840 0x7f71701d04f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 >> 172.21.15.138:0/2959803772 conn(0x7f71700aba10 msgr2=0x7f71700a4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 shutdown_connections 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 -- 172.21.15.138:0/2959803772 wait complete. 2024-09-11T05:20:22.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.458+0000 7f7176416640 1 librados: shutdown 2024-09-11T05:20:22.474 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:20:22.474 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:20:22.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:20:22.475 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:20:22.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.526+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.526+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565447509}) 2024-09-11T05:20:22.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.526+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3741161997}) 2024-09-11T05:20:22.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.526+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.526+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565447509}) 2024-09-11T05:20:22.526 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.526+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=58149480}) 2024-09-11T05:20:22.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206469858}) 2024-09-11T05:20:22.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=191294036}) 2024-09-11T05:20:22.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206469858}) 2024-09-11T05:20:22.528 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373386173}) 2024-09-11T05:20:22.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3561897317}) 2024-09-11T05:20:22.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2918140893}) 2024-09-11T05:20:22.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3561897317}) 2024-09-11T05:20:22.530 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342008857}) 2024-09-11T05:20:22.531 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.530+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3530347605}) 2024-09-11T05:20:22.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3347261935}) 2024-09-11T05:20:22.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3530347605}) 2024-09-11T05:20:22.532 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3423636542}) 2024-09-11T05:20:22.533 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.533 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=768997614}) 2024-09-11T05:20:22.534 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1689095416}) 2024-09-11T05:20:22.534 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.534 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=768997614}) 2024-09-11T05:20:22.534 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3488235221}) 2024-09-11T05:20:22.535 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.535 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.534+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063377201}) 2024-09-11T05:20:22.536 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=272069541}) 2024-09-11T05:20:22.536 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.536 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063377201}) 2024-09-11T05:20:22.536 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036933382}) 2024-09-11T05:20:22.537 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.538 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903166568}) 2024-09-11T05:20:22.538 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2859040952}) 2024-09-11T05:20:22.538 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.538 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903166568}) 2024-09-11T05:20:22.538 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4133719986}) 2024-09-11T05:20:22.539 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.539 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494098038}) 2024-09-11T05:20:22.540 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1461474839}) 2024-09-11T05:20:22.540 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.540 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.538+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494098038}) 2024-09-11T05:20:22.540 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163533493}) 2024-09-11T05:20:22.541 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.541 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150688251}) 2024-09-11T05:20:22.541 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2095214987}) 2024-09-11T05:20:22.542 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.542 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150688251}) 2024-09-11T05:20:22.542 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206210982}) 2024-09-11T05:20:22.543 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.544 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183489581}) 2024-09-11T05:20:22.544 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184773518}) 2024-09-11T05:20:22.544 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.544 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183489581}) 2024-09-11T05:20:22.544 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.542+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902186855}) 2024-09-11T05:20:22.545 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.546+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.546 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.546+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616830656}) 2024-09-11T05:20:22.546 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.546+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2020989865}) 2024-09-11T05:20:22.546 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.546+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.546 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.546+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616830656}) 2024-09-11T05:20:22.546 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.546+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=190860804}) 2024-09-11T05:20:22.547 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.548 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758861477}) 2024-09-11T05:20:22.548 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=941776858}) 2024-09-11T05:20:22.548 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.548 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758861477}) 2024-09-11T05:20:22.548 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4249254409}) 2024-09-11T05:20:22.550 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.550 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203873680}) 2024-09-11T05:20:22.550 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718049919}) 2024-09-11T05:20:22.550 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.550 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203873680}) 2024-09-11T05:20:22.550 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.550+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3308122531}) 2024-09-11T05:20:22.552 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.552 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151485716}) 2024-09-11T05:20:22.552 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1692204513}) 2024-09-11T05:20:22.552 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.552 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151485716}) 2024-09-11T05:20:22.552 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511493825}) 2024-09-11T05:20:22.553 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.554 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2981925229}) 2024-09-11T05:20:22.554 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=474817469}) 2024-09-11T05:20:22.554 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.554 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2981925229}) 2024-09-11T05:20:22.554 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710674884}) 2024-09-11T05:20:22.555 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.555 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174049529}) 2024-09-11T05:20:22.555 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2881394725}) 2024-09-11T05:20:22.556 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.556 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174049529}) 2024-09-11T05:20:22.556 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.554+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3290382159}) 2024-09-11T05:20:22.558 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.558+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.558 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.558+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775336593}) 2024-09-11T05:20:22.558 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.558+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3715302818}) 2024-09-11T05:20:22.558 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.558+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.558 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.558+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775336593}) 2024-09-11T05:20:22.559 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.558+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723694493}) 2024-09-11T05:20:22.560 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.558+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.560 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.558+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197151336}) 2024-09-11T05:20:22.560 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1355786416}) 2024-09-11T05:20:22.560 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.561 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197151336}) 2024-09-11T05:20:22.561 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1698376923}) 2024-09-11T05:20:22.562 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.562 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568944391}) 2024-09-11T05:20:22.562 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420538933}) 2024-09-11T05:20:22.562 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.562 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568944391}) 2024-09-11T05:20:22.562 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3276667515}) 2024-09-11T05:20:22.563 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.564 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122600063}) 2024-09-11T05:20:22.564 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=150747625}) 2024-09-11T05:20:22.564 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.564 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122600063}) 2024-09-11T05:20:22.564 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.562+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2536881656}) 2024-09-11T05:20:22.566 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.566+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.566 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.566+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250777910}) 2024-09-11T05:20:22.566 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.566+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1306615406}) 2024-09-11T05:20:22.566 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.566+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.566 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.566+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250777910}) 2024-09-11T05:20:22.566 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.566+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2070427340}) 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient: get_monmap_and_config 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient: build_initial_monmap 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient: monmap: 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:22.609656+0000 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:22.609656+0000 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:22.605 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 1 Processor -- start 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 1 -- start start 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient: init 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f374c063d50 addr v1:172.21.15.138:6789/0 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f374c06fcf0 0x7f374c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient(hunting): picked mon.noname-c con 0x7f374c06fcf0 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient(hunting): picked mon.noname-b con 0x7f374c13dc00 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374c06ef80 con 0x7f374c063d50 2024-09-11T05:20:22.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f374c0a0ec0 con 0x7f374c06fcf0 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f374c064130 con 0x7f374c13dc00 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3751dea640 10 monclient(hunting): authenticate will time out at 1706.045771s 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374b7fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f374c063d50 0x7f374c0a4a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:32812/0 (socket says 172.21.15.138:32812) 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374b7fe640 1 -- 172.21.15.138:0/784452124 learned_addr learned my addr 172.21.15.138:0/784452124 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 1 --2- 172.21.15.138:0/784452124 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient(hunting): get_auth_request con 0x7f374c13dc00 auth_method 0 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3742ffd640 1 --2- 172.21.15.138:0/784452124 >> v2:172.21.15.190:3300/0 conn(0x7f374c06fcf0 0x7f374c0a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3742ffd640 10 monclient(hunting): get_auth_request con 0x7f374c06fcf0 auth_method 0 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3742ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient(hunting): handle_auth_done global_id 84323 payload 274 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 1 -- 172.21.15.138:0/784452124 >> v2:172.21.15.190:3300/0 conn(0x7f374c06fcf0 msgr2=0x7f374c0a5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 1 --2- 172.21.15.138:0/784452124 >> v2:172.21.15.190:3300/0 conn(0x7f374c06fcf0 0x7f374c0a5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 1 -- 172.21.15.138:0/784452124 >> v1:172.21.15.138:6789/0 conn(0x7f374c063d50 legacy=0x7f374c0a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 1 monclient: found mon.noname-b 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:20:22.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f374bfff640 1 -- 172.21.15.138:0/784452124 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c06f470 con 0x7f374c13dc00 2024-09-11T05:20:22.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.606+0000 7f3742ffd640 1 --2- 172.21.15.138:0/784452124 >> v2:172.21.15.190:3300/0 conn(0x7f374c06fcf0 0x7f374c0a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:20:22.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374bfff640 1 -- 172.21.15.138:0/784452124 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 msgr2=0x7f374c13fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374bfff640 1 --2- 172.21.15.138:0/784452124 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c13fff0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3744009d90 tx=0x7f374401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8a60eef79a83e09 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 1 -- 172.21.15.138:0/784452124 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37440b2070 con 0x7f374c13dc00 2024-09-11T05:20:22.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: dump: 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: _finish_auth 0 2024-09-11T05:20:22.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: _check_auth_tickets 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 5 monclient: authenticate success, global_id 84323 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 1 -- 172.21.15.138:0/784452124 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37440a5760 con 0x7f374c13dc00 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 4 set_mon_vals no callback set 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 1 -- 172.21.15.138:0/784452124 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37440a5b30 con 0x7f374c13dc00 2024-09-11T05:20:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f374affd640 10 monclient: dump: 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.620 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 10 monclient: shutdown 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 1 -- 172.21.15.138:0/784452124 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 msgr2=0x7f374c13fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 1 --2- 172.21.15.138:0/784452124 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c13fff0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3744009d90 tx=0x7f374401c830 comp rx=0 tx=0).stop 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 1 -- 172.21.15.138:0/784452124 shutdown_connections 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 1 --2- 172.21.15.138:0/784452124 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c13fff0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 1 --2- 172.21.15.138:0/784452124 >> v2:172.21.15.190:3300/0 conn(0x7f374c06fcf0 0x7f374c0a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 1 -- 172.21.15.138:0/784452124 >> 172.21.15.138:0/784452124 conn(0x7f374c0a6180 msgr2=0x7f374c063640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 1 -- 172.21.15.138:0/784452124 shutdown_connections 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.610+0000 7f3751dea640 1 -- 172.21.15.138:0/784452124 wait complete. 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient: build_initial_monmap 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient: monmap: 2024-09-11T05:20:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 librados: starting msgr at 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 librados: starting objecter 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 Processor -- start 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 -- start start 2024-09-11T05:20:22.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 librados: setting wanted keys 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 librados: calling monclient init 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient: init 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c063d50 0x7f374c125f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient(hunting): picked mon.noname-b con 0x7f374c063d50 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c06fcf0 0x7f374c13cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient(hunting): picked mon.noname-a con 0x7f374c06fcf0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c1d0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient(hunting): picked mon.noname-c con 0x7f374c13dc00 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f374c142ed0 con 0x7f374c06fcf0 2024-09-11T05:20:22.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f374c142d50 con 0x7f374c063d50 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f374c143050 con 0x7f374c13dc00 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient(hunting): authenticate will time out at 1706.053022s 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c1d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c06fcf0 0x7f374c13cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c1d0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48024/0 (socket says 172.21.15.138:48024) 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 1 -- 172.21.15.138:0/169069178 learned_addr learned my addr 172.21.15.138:0/169069178 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 10 monclient(hunting): get_auth_request con 0x7f374c13dc00 auth_method 0 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient(hunting): get_auth_request con 0x7f374c06fcf0 auth_method 0 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374b7fe640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c063d50 0x7f374c125f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374b7fe640 10 monclient(hunting): get_auth_request con 0x7f374c063d50 auth_method 0 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient(hunting): handle_auth_done global_id 64641 payload 824 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 msgr2=0x7f374c1d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c1d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c063d50 msgr2=0x7f374c125f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f374c063d50 0x7f374c125f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 monclient: found mon.noname-a 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 -- 172.21.15.138:0/169069178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c125400 con 0x7f374c06fcf0 2024-09-11T05:20:22.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374bfff640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f374c13dc00 0x7f374c1d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c06fcf0 msgr2=0x7f374c13cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3742ffd640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c06fcf0 0x7f374c13cdd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f373400e7e0 tx=0x7f373400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 client.?.objecter ms_handle_connect 0x7f374c06fcf0 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734017070 con 0x7f374c06fcf0 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: dump: 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.627 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: _finish_auth 0 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373400c8d0 con 0x7f374c06fcf0 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373400f040 con 0x7f374c06fcf0 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 5 monclient: authenticate success, global_id 64641 2024-09-11T05:20:22.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient: _renew_subs 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f374a7fc640 4 set_mon_vals no callback set 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374c0a0210 con 0x7f374c06fcf0 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.614+0000 7f3748ff9640 10 monclient: dump: 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.629 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 10 client.64641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 10 monclient: _renew_subs 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f374c127760 con 0x7f374c06fcf0 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 1 librados: init done 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c13d310 0x7f374c1d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f374c0a8930 con 0x7f374c13d310 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c13d310 0x7f374c1d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 10 monclient: get_auth_request con 0x7f374c13d310 auth_method 0 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3734012050 con 0x7f374c06fcf0 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 10 monclient(hunting): handle_auth_done global_id 64644 payload 824 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 1 -- 172.21.15.138:0/169069178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f374c0a4a30 con 0x7f374c13d310 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 1 --2- 172.21.15.138:0/169069178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f372c03ced0 0x7f372c03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7f373404f8a0 con 0x7f374c06fcf0 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 10 client.64641.objecter ms_dispatch 0x7f374c000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 3 client.64641.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 3 client.64641.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:22.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 20 client.64641.objecter dump_active .. 0 homeless 2024-09-11T05:20:22.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374bfff640 1 --2- 172.21.15.138:0/169069178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f372c03ced0 0x7f372c03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374bfff640 10 monclient: get_auth_request con 0x7f372c03ced0 auth_method 0 2024-09-11T05:20:22.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 10 client.64641.objecter ms_handle_connect 0x7f374c13d310 2024-09-11T05:20:22.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 10 client.64641.objecter resend_mon_ops 2024-09-11T05:20:22.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c13d310 msgr2=0x7f374c1d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f374b7fe640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c13d310 0x7f374c1d07d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f373c00e460 tx=0x7f373c00e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c005d00 con 0x7f374c13d310 2024-09-11T05:20:22.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 10 monclient: dump: 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f373c018070 con 0x7f374c13d310 2024-09-11T05:20:22.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.618+0000 7f3748ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.622+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c13d310 msgr2=0x7f374c1d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.622+0000 7f3748ff9640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c13d310 0x7f374c1d07d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f373c00e460 tx=0x7f373c00e920 comp rx=0 tx=0).stop 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.622+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c13d310 msgr2=0x7f374c1d07d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.622+0000 7f3748ff9640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c13d310 0x7f374c1d07d0 secure :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f373c00e460 tx=0x7f373c00e920 comp rx=0 tx=0).stop 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.622+0000 7f374bfff640 1 --2- 172.21.15.138:0/169069178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f372c03ced0 0x7f372c03f390 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f374401c660 tx=0x7f37440a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.622+0000 7f3748ff9640 10 client.64641.objecter ms_handle_connect 0x7f372c03ced0 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f3751dea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f3751dea640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c123ed0 0x7f374c1d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f3751dea640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f374c06cbf0 con 0x7f374c123ed0 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c123ed0 0x7f374c1d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 10 monclient: get_auth_request con 0x7f374c123ed0 auth_method 0 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 10 monclient(hunting): handle_auth_done global_id 64647 payload 824 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.626+0000 7f374b7fe640 1 -- 172.21.15.138:0/169069178 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f374c1db620 con 0x7f374c123ed0 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 10 client.64641.objecter ms_handle_connect 0x7f374c123ed0 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f374b7fe640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c123ed0 msgr2=0x7f374c1d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 10 client.64641.objecter resend_mon_ops 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f374b7fe640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c123ed0 0x7f374c1d0d10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f373c017000 tx=0x7f373c004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c0074e0 con 0x7f374c123ed0 2024-09-11T05:20:22.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 10 monclient: dump: 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f373c0074e0 con 0x7f374c123ed0 2024-09-11T05:20:22.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c123ed0 msgr2=0x7f374c1d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c123ed0 0x7f374c1d0d10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f373c017000 tx=0x7f373c004950 comp rx=0 tx=0).stop 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c123ed0 msgr2=0x7f374c1d0d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3748ff9640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c123ed0 0x7f374c1d0d10 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f373c017000 tx=0x7f373c004950 comp rx=0 tx=0).stop 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:20:22.639 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 52, 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 526260, 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 6, 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.640 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.641 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:20:22.642 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:20:22.643 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.644 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:20:22.649 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:22.650 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:20:22.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:20:22.652 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:22.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 10 librados: watch_flush enter 2024-09-11T05:20:22.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 10 librados: watch_flush exit 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 20 client.64641.objecter shutdown clearing up homeless session... 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 10 client.64641.objecter successfully canceled tick 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f372c03ced0 msgr2=0x7f372c03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 --2- 172.21.15.138:0/169069178 >> v2:172.21.15.190:6800/1562782189 conn(0x7f372c03ced0 0x7f372c03f390 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f374401c660 tx=0x7f37440a7070 comp rx=0 tx=0).stop 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 10 monclient: shutdown 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c06fcf0 msgr2=0x7f374c13cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c06fcf0 0x7f374c13cdd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f373400e7e0 tx=0x7f373400eca0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f374bfff640 1 -- 172.21.15.138:0/169069178 reap_dead start 2024-09-11T05:20:22.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 shutdown_connections 2024-09-11T05:20:22.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 --2- 172.21.15.138:0/169069178 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f374c06fcf0 0x7f374c13cdd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 >> 172.21.15.138:0/169069178 conn(0x7f374c0a6180 msgr2=0x7f374c0a6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:22.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 shutdown_connections 2024-09-11T05:20:22.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 -- 172.21.15.138:0/169069178 wait complete. 2024-09-11T05:20:22.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.630+0000 7f3751dea640 1 librados: shutdown 2024-09-11T05:20:22.656 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:20:22.656 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:20:22.656 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient: get_monmap_and_config 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient: build_initial_monmap 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient: monmap: 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:22.773827+0000 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:22.773827+0000 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:22.770 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 1 Processor -- start 2024-09-11T05:20:22.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 1 -- start start 2024-09-11T05:20:22.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient: init 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd9c073e80 0x7fdd9c0a2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9c073e80 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074260 0x7fdd9c0a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient(hunting): picked mon.noname-b con 0x7fdd9c074260 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdd9c074640 0x7fdd9c0a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient(hunting): picked mon.noname-a con 0x7fdd9c074640 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fdd9c0a1ea0 con 0x7fdd9c074640 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdd9c071a10 con 0x7fdd9c073e80 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9c0a2b40 con 0x7fdd9c074260 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda2718640 10 monclient(hunting): authenticate will time out at 1706.209933s 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdd9c074640 0x7fdd9c0a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074260 0x7fdd9c0a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdd9c074640 0x7fdd9c0a3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34360/0 (socket says 172.21.15.138:34360) 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074260 0x7fdd9c0a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48036/0 (socket says 172.21.15.138:48036) 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 -- 172.21.15.138:0/742900829 learned_addr learned my addr 172.21.15.138:0/742900829 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient(hunting): get_auth_request con 0x7fdd9c074640 auth_method 0 2024-09-11T05:20:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 10 monclient(hunting): get_auth_request con 0x7fdd9c074260 auth_method 0 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd9bfff640 1 --2- 172.21.15.138:0/742900829 >> v2:172.21.15.190:3300/0 conn(0x7fdd9c073e80 0x7fdd9c0a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd9bfff640 10 monclient(hunting): get_auth_request con 0x7fdd9c073e80 auth_method 0 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient(hunting): handle_auth_done global_id 64650 payload 274 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 -- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074260 msgr2=0x7fdd9c0a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 --2- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074260 0x7fdd9c0a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 -- 172.21.15.138:0/742900829 >> v2:172.21.15.190:3300/0 conn(0x7fdd9c073e80 msgr2=0x7fdd9c0a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 --2- 172.21.15.138:0/742900829 >> v2:172.21.15.190:3300/0 conn(0x7fdd9c073e80 0x7fdd9c0a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient: _finish_hunting 0 2024-09-11T05:20:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 monclient: found mon.noname-a 2024-09-11T05:20:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdda0cb5640 1 -- 172.21.15.138:0/742900829 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9c0a0560 con 0x7fdd9c074640 2024-09-11T05:20:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.770+0000 7fdd937fe640 1 --2- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074260 0x7fdd9c0a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:22.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda0cb5640 1 -- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c074640 msgr2=0x7fdd9c0a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda0cb5640 1 --2- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c074640 0x7fdd9c0a3e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0a6a40 tx=0x7fdd9401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0494425f310ed98 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 1 -- 172.21.15.138:0/742900829 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940a5530 con 0x7fdd9c074640 2024-09-11T05:20:22.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: dump: 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:20:22.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 1 -- 172.21.15.138:0/742900829 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd940a56d0 con 0x7fdd9c074640 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 5 monclient: authenticate success, global_id 64650 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 1 -- 172.21.15.138:0/742900829 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940a5aa0 con 0x7fdd9c074640 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 4 set_mon_vals no callback set 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:22.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdd9b7fe640 10 monclient: dump: 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.789 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 10 monclient: shutdown 2024-09-11T05:20:22.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:22.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 -- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c074640 msgr2=0x7fdd9c0a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 --2- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c074640 0x7fdd9c0a3e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0a6a40 tx=0x7fdd9401c8b0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 -- 172.21.15.138:0/742900829 shutdown_connections 2024-09-11T05:20:22.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 --2- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c074640 0x7fdd9c0a3e10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 --2- 172.21.15.138:0/742900829 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074260 0x7fdd9c0a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 --2- 172.21.15.138:0/742900829 >> v2:172.21.15.190:3300/0 conn(0x7fdd9c073e80 0x7fdd9c0a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.790 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 -- 172.21.15.138:0/742900829 >> 172.21.15.138:0/742900829 conn(0x7fdd9c0a8ed0 msgr2=0x7fdd9c0a92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 -- 172.21.15.138:0/742900829 shutdown_connections 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.774+0000 7fdda2718640 1 -- 172.21.15.138:0/742900829 wait complete. 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient: build_initial_monmap 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient: monmap: 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:22.791 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 librados: starting msgr at 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 librados: starting objecter 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 Processor -- start 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 -- start start 2024-09-11T05:20:22.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 librados: setting wanted keys 2024-09-11T05:20:22.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 librados: calling monclient init 2024-09-11T05:20:22.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient: init 2024-09-11T05:20:22.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:22.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:20:22.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c073e80 0x7fdd9c1ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient(hunting): picked mon.noname-a con 0x7fdd9c073e80 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:22.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd9c074260 0x7fdd9c13c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient(hunting): picked mon.noname-b con 0x7fdd9c074260 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074640 0x7fdd9c124230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9c074640 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c140320 con 0x7fdd9c073e80 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c1401a0 con 0x7fdd9c074260 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9c1404a0 con 0x7fdd9c074640 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda2718640 10 monclient(hunting): authenticate will time out at 1706.217363s 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c073e80 0x7fdd9c1ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074640 0x7fdd9c124230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c073e80 0x7fdd9c1ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34364/0 (socket says 172.21.15.138:34364) 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 -- 172.21.15.138:0/3745837783 learned_addr learned my addr 172.21.15.138:0/3745837783 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient(hunting): get_auth_request con 0x7fdd9c073e80 auth_method 0 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074640 0x7fdd9c124230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48048/0 (socket says 172.21.15.138:48048) 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 10 monclient(hunting): get_auth_request con 0x7fdd9c074640 auth_method 0 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdda0cb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd937fe640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd9c074260 0x7fdd9c13c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient(hunting): handle_auth_done global_id 64653 payload 824 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074640 msgr2=0x7fdd9c124230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c074640 0x7fdd9c124230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd9c074260 msgr2=0x7fdd9c13c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd9c074260 0x7fdd9c13c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 monclient: found mon.noname-a 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd9bfff640 1 -- 172.21.15.138:0/3745837783 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9c128050 con 0x7fdd9c073e80 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd937fe640 10 monclient: get_auth_request con 0x7fdd9c074260 auth_method 0 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.778+0000 7fdd937fe640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd9c074260 0x7fdd9c13c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd9bfff640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c073e80 msgr2=0x7fdd9c1ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd9bfff640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c073e80 0x7fdd9c1ce6c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00d800 tx=0x7fdd8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 client.?.objecter ms_handle_connect 0x7fdd9c073e80 2024-09-11T05:20:22.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00b8d0 con 0x7fdd9c073e80 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: dump: 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.798 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: _finish_auth 0 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: _check_auth_tickets 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd8c019070 con 0x7fdd9c073e80 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 5 monclient: authenticate success, global_id 64653 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 10 monclient: _renew_subs 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9c0a00c0 con 0x7fdd9c073e80 2024-09-11T05:20:22.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00bc70 con 0x7fdd9c073e80 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: dump: 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd9a7fc640 4 set_mon_vals no callback set 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 10 client.64653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 10 monclient: _renew_subs 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd9c127d70 con 0x7fdd9c073e80 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 1 librados: init done 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c13cbf0 0x7fdd9c02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9c0a2600 con 0x7fdd9c13cbf0 2024-09-11T05:20:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c13cbf0 0x7fdd9c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 10 monclient: get_auth_request con 0x7fdd9c13cbf0 auth_method 0 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdd8c013070 con 0x7fdd9c073e80 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 10 monclient(hunting): handle_auth_done global_id 84335 payload 824 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 1 -- 172.21.15.138:0/3745837783 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd9c074a40 con 0x7fdd9c13cbf0 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 1 --2- 172.21.15.138:0/3745837783 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd7c03cea0 0x7fdd7c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7fdd8c04fd30 con 0x7fdd9c073e80 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 client.64653.objecter ms_dispatch 0x7fdd9c000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 3 client.64653.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 3 client.64653.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 20 client.64653.objecter dump_active .. 0 homeless 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda0cb5640 1 --2- 172.21.15.138:0/3745837783 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd7c03cea0 0x7fdd7c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdda0cb5640 10 monclient: get_auth_request con 0x7fdd7c03cea0 auth_method 0 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 client.64653.objecter ms_handle_connect 0x7fdd9c13cbf0 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 client.64653.objecter resend_mon_ops 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c13cbf0 msgr2=0x7fdd9c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd937fe640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c13cbf0 0x7fdd9c02d450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd840107c0 tx=0x7fdd84010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8400f040 con 0x7fdd9c13cbf0 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:22.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.782+0000 7fdd997fa640 10 monclient: dump: 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdd8401c400 con 0x7fdd9c13cbf0 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdd997fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdd997fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:22.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c13cbf0 msgr2=0x7fdd9c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdd997fa640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c13cbf0 0x7fdd9c02d450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd840107c0 tx=0x7fdd84010c80 comp rx=0 tx=0).stop 2024-09-11T05:20:22.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c13cbf0 msgr2=0x7fdd9c02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:22.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdd997fa640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c13cbf0 0x7fdd9c02d450 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd840107c0 tx=0x7fdd84010c80 comp rx=0 tx=0).stop 2024-09-11T05:20:22.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdda0cb5640 1 --2- 172.21.15.138:0/3745837783 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd7c03cea0 0x7fdd7c03f360 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0a2350 tx=0x7fdd940a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.786+0000 7fdd997fa640 10 client.64653.objecter ms_handle_connect 0x7fdd7c03cea0 2024-09-11T05:20:22.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdda2718640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdda2718640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c02d990 0x7fdd9c02dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdda2718640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdd9c127940 con 0x7fdd9c02d990 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c02d990 0x7fdd9c02dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 10 monclient: get_auth_request con 0x7fdd9c02d990 auth_method 0 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 10 monclient(hunting): handle_auth_done global_id 84338 payload 824 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.790+0000 7fdd937fe640 1 -- 172.21.15.138:0/3745837783 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdd9c120c30 con 0x7fdd9c02d990 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd937fe640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c02d990 msgr2=0x7fdd9c02dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd937fe640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c02d990 0x7fdd9c02dda0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd84011010 tx=0x7fdd840046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 10 client.64653.objecter ms_handle_connect 0x7fdd9c02d990 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 10 client.64653.objecter resend_mon_ops 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84005db0 con 0x7fdd9c02d990 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 10 monclient: dump: 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.808 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7fdd84025070 con 0x7fdd9c02d990 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c02d990 msgr2=0x7fdd9c02dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c02d990 0x7fdd9c02dda0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd84011010 tx=0x7fdd840046d0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c02d990 msgr2=0x7fdd9c02dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:22.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdd997fa640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd9c02d990 0x7fdd9c02dda0 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd84011010 tx=0x7fdd840046d0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 52, 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 266564, 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:20:22.810 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 6, 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.811 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.812 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:22.813 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.814 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.815 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.816 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:20:22.818 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:20:22.819 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.819 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.819 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.819 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:22.819 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:22.820 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.820 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.820 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.820 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:20:22.820 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.821 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.821 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:20:22.821 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.821 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.821 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:20:22.822 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.824 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:20:22.824 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:22.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 10 librados: watch_flush enter 2024-09-11T05:20:22.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 10 librados: watch_flush exit 2024-09-11T05:20:22.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 20 client.64653.objecter shutdown clearing up homeless session... 2024-09-11T05:20:22.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 10 client.64653.objecter successfully canceled tick 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd7c03cea0 msgr2=0x7fdd7c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 --2- 172.21.15.138:0/3745837783 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdd7c03cea0 0x7fdd7c03f360 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0a2350 tx=0x7fdd940a7070 comp rx=0 tx=0).stop 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 10 monclient: shutdown 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c073e80 msgr2=0x7fdd9c1ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c073e80 0x7fdd9c1ce6c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00d800 tx=0x7fdd8c00dcc0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda0cb5640 1 -- 172.21.15.138:0/3745837783 reap_dead start 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 shutdown_connections 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 --2- 172.21.15.138:0/3745837783 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd9c073e80 0x7fdd9c1ce6c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 >> 172.21.15.138:0/3745837783 conn(0x7fdd9c0a8ed0 msgr2=0x7fdd9c09ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:22.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 shutdown_connections 2024-09-11T05:20:22.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 -- 172.21.15.138:0/3745837783 wait complete. 2024-09-11T05:20:22.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.794+0000 7fdda2718640 1 librados: shutdown 2024-09-11T05:20:22.827 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:20:22.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:20:22.827 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:20:22.895 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.894+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.895 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.894+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283055592}) 2024-09-11T05:20:22.895 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.894+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3339183740}) 2024-09-11T05:20:22.895 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.894+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.895 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.894+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283055592}) 2024-09-11T05:20:22.895 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.894+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116856268}) 2024-09-11T05:20:22.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.942+0000 7f6023ae9640 10 monclient: get_monmap_and_config 2024-09-11T05:20:22.941 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.942+0000 7f6023ae9640 10 monclient: build_initial_monmap 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.942+0000 7f6023ae9640 10 monclient: monmap: 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:22.947299+0000 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:22.947299+0000 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:22.942 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.942+0000 7f6023ae9640 1 Processor -- start 2024-09-11T05:20:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.942+0000 7f6023ae9640 1 -- start start 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.942+0000 7f6023ae9640 10 monclient: init 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f601c070010 0x7f601c135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient(hunting): picked mon.noname-c con 0x7f601c070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c09e6e0 0x7f601c09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient(hunting): picked mon.noname-b con 0x7f601c09e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f601c1382c0 addr v1:172.21.15.138:6789/0 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c0a8990 con 0x7f601c1382c0 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f601c0703f0 con 0x7f601c070010 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f601c09cbf0 con 0x7f601c09e6e0 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6023ae9640 10 monclient(hunting): authenticate will time out at 1706.384231s 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c09e6e0 0x7f601c09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6022086640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f601c1382c0 0x7f601c138770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:32822/0 (socket says 172.21.15.138:32822) 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6022086640 1 -- 172.21.15.138:0/928627143 learned_addr learned my addr 172.21.15.138:0/928627143 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient(hunting): get_auth_request con 0x7f601c09e6e0 auth_method 0 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021885640 1 --2- 172.21.15.138:0/928627143 >> v2:172.21.15.190:3300/0 conn(0x7f601c070010 0x7f601c135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.947 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021885640 10 monclient(hunting): get_auth_request con 0x7f601c070010 auth_method 0 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021885640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021885640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient(hunting): handle_auth_done global_id 84341 payload 274 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 1 -- 172.21.15.138:0/928627143 >> v2:172.21.15.190:3300/0 conn(0x7f601c070010 msgr2=0x7f601c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 1 --2- 172.21.15.138:0/928627143 >> v2:172.21.15.190:3300/0 conn(0x7f601c070010 0x7f601c135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 1 -- 172.21.15.138:0/928627143 >> v1:172.21.15.138:6789/0 conn(0x7f601c1382c0 legacy=0x7f601c138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient: _finish_hunting 0 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 1 monclient: found mon.noname-b 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:20:22.948 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.946+0000 7f6021084640 1 -- 172.21.15.138:0/928627143 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c09ced0 con 0x7f601c09e6e0 2024-09-11T05:20:22.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021084640 1 -- 172.21.15.138:0/928627143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c09e6e0 msgr2=0x7f601c09eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021084640 1 --2- 172.21.15.138:0/928627143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c09e6e0 0x7f601c09eae0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f600c0097c0 tx=0x7f600c01c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=981bea99c1210616 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 1 -- 172.21.15.138:0/928627143 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c0b0070 con 0x7f601c09e6e0 2024-09-11T05:20:22.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:20:22.953 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: dump: 2024-09-11T05:20:22.953 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.953 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: _finish_auth 0 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: _check_auth_tickets 2024-09-11T05:20:22.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 1 -- 172.21.15.138:0/928627143 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f600c0ab600 con 0x7f601c09e6e0 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 1 -- 172.21.15.138:0/928627143 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c0ab9d0 con 0x7f601c09e6e0 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 5 monclient: authenticate success, global_id 84341 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 4 set_mon_vals no callback set 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6020883640 10 monclient: dump: 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.955 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient: shutdown 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- 172.21.15.138:0/928627143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c09e6e0 msgr2=0x7f601c09eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 --2- 172.21.15.138:0/928627143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c09e6e0 0x7f601c09eae0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f600c0097c0 tx=0x7f600c01c330 comp rx=0 tx=0).stop 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- 172.21.15.138:0/928627143 shutdown_connections 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 --2- 172.21.15.138:0/928627143 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c09e6e0 0x7f601c09eae0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 --2- 172.21.15.138:0/928627143 >> v2:172.21.15.190:3300/0 conn(0x7f601c070010 0x7f601c135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- 172.21.15.138:0/928627143 >> 172.21.15.138:0/928627143 conn(0x7f601c0ab830 msgr2=0x7f601c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- 172.21.15.138:0/928627143 shutdown_connections 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- 172.21.15.138:0/928627143 wait complete. 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient: build_initial_monmap 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient: monmap: 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:22.957 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 librados: starting msgr at 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 librados: starting objecter 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 Processor -- start 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- start start 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 librados: setting wanted keys 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 librados: calling monclient init 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient: init 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c070010 0x7f601c1d0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient(hunting): picked mon.noname-c con 0x7f601c070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c09e6e0 0x7f601c1d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.958 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient(hunting): picked mon.noname-b con 0x7f601c09e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f601c1382c0 0x7f601c1d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient(hunting): picked mon.noname-a con 0x7f601c1382c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f601c1432f0 con 0x7f601c1382c0 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f601c143170 con 0x7f601c09e6e0 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f601c143470 con 0x7f601c070010 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6023ae9640 10 monclient(hunting): authenticate will time out at 1706.391506s 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c070010 0x7f601c1d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c070010 0x7f601c1d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48084/0 (socket says 172.21.15.138:48084) 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 1 -- 172.21.15.138:0/2307165277 learned_addr learned my addr 172.21.15.138:0/2307165277 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:22.959 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 10 monclient(hunting): get_auth_request con 0x7f601c070010 auth_method 0 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f601c1382c0 0x7f601c1d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient(hunting): get_auth_request con 0x7f601c1382c0 auth_method 0 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021084640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c09e6e0 0x7f601c1d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient(hunting): handle_auth_done global_id 64656 payload 824 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c070010 msgr2=0x7f601c1d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c070010 0x7f601c1d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c09e6e0 msgr2=0x7f601c1d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c09e6e0 0x7f601c1d0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient: _finish_hunting 0 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 1 monclient: found mon.noname-a 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6022086640 1 -- 172.21.15.138:0/2307165277 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c0a3d70 con 0x7f601c1382c0 2024-09-11T05:20:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021084640 10 monclient: get_auth_request con 0x7f601c09e6e0 auth_method 0 2024-09-11T05:20:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.950+0000 7f6021885640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f601c070010 0x7f601c1d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6021084640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c09e6e0 0x7f601c1d0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:20:22.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6022086640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f601c1382c0 msgr2=0x7f601c1d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6022086640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f601c1382c0 0x7f601c1d11c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f601400caf0 tx=0x7f601400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 client.?.objecter ms_handle_connect 0x7f601c1382c0 2024-09-11T05:20:22.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:22.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601400ade0 con 0x7f601c1382c0 2024-09-11T05:20:22.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: dump: 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: _finish_auth 0 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: _check_auth_tickets 2024-09-11T05:20:22.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 5 monclient: authenticate success, global_id 64656 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6014012400 con 0x7f601c1382c0 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140127f0 con 0x7f601c1382c0 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 10 monclient: _renew_subs 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601c0a3a90 con 0x7f601c1382c0 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6013fff640 4 set_mon_vals no callback set 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 monclient: dump: 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.969 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 10 client.64656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 10 monclient: _renew_subs 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601c0a4ab0 con 0x7f601c1382c0 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 1 librados: init done 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0a3fb0 0x7f601c12e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f601c06fd00 con 0x7f601c0a3fb0 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f60140129d0 con 0x7f601c1382c0 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6021885640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0a3fb0 0x7f601c12e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 1 --2- 172.21.15.138:0/2307165277 >> v2:172.21.15.190:6800/1562782189 conn(0x7f600003cef0 0x7f600003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6021885640 10 monclient: get_auth_request con 0x7f601c0a3fb0 auth_method 0 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6021885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6021885640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6021885640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7f6014050190 con 0x7f601c1382c0 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 10 client.64656.objecter ms_dispatch 0x7f601c000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 3 client.64656.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f60127fc640 3 client.64656.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.954+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 20 client.64656.objecter dump_active .. 0 homeless 2024-09-11T05:20:22.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f6021084640 1 --2- 172.21.15.138:0/2307165277 >> v2:172.21.15.190:6800/1562782189 conn(0x7f600003cef0 0x7f600003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f6021084640 10 monclient: get_auth_request con 0x7f600003cef0 auth_method 0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f6021885640 10 monclient(hunting): handle_auth_done global_id 104119 payload 824 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f6021885640 1 -- 172.21.15.138:0/2307165277 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f601c135450 con 0x7f601c0a3fb0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 10 client.64656.objecter ms_handle_connect 0x7f601c0a3fb0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f6021885640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0a3fb0 msgr2=0x7f601c12e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 10 client.64656.objecter resend_mon_ops 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f6021885640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0a3fb0 0x7f601c12e860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6018010cb0 tx=0x7f601800e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 10 client.64656.objecter ms_handle_connect 0x7f600003cef0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f6021084640 1 --2- 172.21.15.138:0/2307165277 >> v2:172.21.15.190:6800/1562782189 conn(0x7f600003cef0 0x7f600003f3b0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f600c0ab000 tx=0x7f600c0a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6018018070 con 0x7f601c0a3fb0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 10 monclient: dump: 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6018014640 con 0x7f601c0a3fb0 2024-09-11T05:20:22.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0a3fb0 msgr2=0x7f601c12e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.958+0000 7f60127fc640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0a3fb0 0x7f601c12e860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6018010cb0 tx=0x7f601800e5a0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.962+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0a3fb0 msgr2=0x7f601c12e860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.962+0000 7f60127fc640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c0a3fb0 0x7f601c12e860 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6018010cb0 tx=0x7f601800e5a0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6023ae9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6023ae9640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c12eda0 0x7f601c12f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6023ae9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f601c12f6f0 con 0x7f601c12eda0 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c12eda0 0x7f601c12f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 10 monclient: get_auth_request con 0x7f601c12eda0 auth_method 0 2024-09-11T05:20:22.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:22.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:22.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:22.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:22.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:22.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:22.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 10 monclient(hunting): handle_auth_done global_id 104122 payload 824 2024-09-11T05:20:22.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.966+0000 7f6021885640 1 -- 172.21.15.138:0/2307165277 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f601c069fa0 con 0x7f601c12eda0 2024-09-11T05:20:22.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6021885640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c12eda0 msgr2=0x7f601c12f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:22.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 10 client.64656.objecter ms_handle_connect 0x7f601c12eda0 2024-09-11T05:20:22.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6021885640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c12eda0 0x7f601c12f1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6018010cb0 tx=0x7f60180046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 10 client.64656.objecter resend_mon_ops 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601801daf0 con 0x7f601c12eda0 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 10 monclient: dump: 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:22.977 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f60180273f0 con 0x7f601c12eda0 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c12eda0 msgr2=0x7f601c12f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c12eda0 0x7f601c12f1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6018010cb0 tx=0x7f60180046d0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c12eda0 msgr2=0x7f601c12f1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f60127fc640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f601c12eda0 0x7f601c12f1b0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6018010cb0 tx=0x7f60180046d0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:22.978 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 52, 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 13135, 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 6, 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.979 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:20:22.980 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:20:22.981 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:20:22.982 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:22.983 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.984 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.986 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.987 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:22.988 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:22.990 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:22.991 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:22.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:22.991 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:20:22.991 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:22.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296853866}) 2024-09-11T05:20:22.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171013177}) 2024-09-11T05:20:22.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:22.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 10 librados: watch_flush enter 2024-09-11T05:20:22.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296853866}) 2024-09-11T05:20:22.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2477975058}) 2024-09-11T05:20:22.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 10 librados: watch_flush exit 2024-09-11T05:20:22.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 20 client.64656.objecter shutdown clearing up homeless session... 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 10 client.64656.objecter successfully canceled tick 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 >> v2:172.21.15.190:6800/1562782189 conn(0x7f600003cef0 msgr2=0x7f600003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 --2- 172.21.15.138:0/2307165277 >> v2:172.21.15.190:6800/1562782189 conn(0x7f600003cef0 0x7f600003f3b0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f600c0ab000 tx=0x7f600c0a6000 comp rx=0 tx=0).stop 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 10 monclient: shutdown 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f601c1382c0 msgr2=0x7f601c1d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6022086640 1 -- 172.21.15.138:0/2307165277 reap_dead start 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f601c1382c0 0x7f601c1d11c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f601400caf0 tx=0x7f601400cef0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 shutdown_connections 2024-09-11T05:20:22.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 --2- 172.21.15.138:0/2307165277 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f601c1382c0 0x7f601c1d11c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:22.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 >> 172.21.15.138:0/2307165277 conn(0x7f601c0ab830 msgr2=0x7f601c09d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:22.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 shutdown_connections 2024-09-11T05:20:22.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 -- 172.21.15.138:0/2307165277 wait complete. 2024-09-11T05:20:22.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:22.970+0000 7f6023ae9640 1 librados: shutdown 2024-09-11T05:20:22.996 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:20:22.996 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T05:20:22.997 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T05:20:22.997 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T05:20:22.997 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-11T05:20:22.997 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-11T05:20:23.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:23.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4045930802}) 2024-09-11T05:20:23.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195000309}) 2024-09-11T05:20:23.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:23.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4045930802}) 2024-09-11T05:20:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2997314185}) 2024-09-11T05:20:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258493312}) 2024-09-11T05:20:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751547617}) 2024-09-11T05:20:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258493312}) 2024-09-11T05:20:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.022+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=987380869}) 2024-09-11T05:20:23.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:23.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031133553}) 2024-09-11T05:20:23.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1861281044}) 2024-09-11T05:20:23.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:23.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031133553}) 2024-09-11T05:20:23.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=496618418}) 2024-09-11T05:20:23.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:23.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643033935}) 2024-09-11T05:20:23.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851832875}) 2024-09-11T05:20:23.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:20:23.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643033935}) 2024-09-11T05:20:23.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:20:23.026+0000 7f1fc4bd5640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1327583273}) 2024-09-11T05:20:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient: get_monmap_and_config 2024-09-11T05:20:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient: build_initial_monmap 2024-09-11T05:20:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient: monmap: 2024-09-11T05:20:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:23.119266+0000 2024-09-11T05:20:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:23.119266+0000 2024-09-11T05:20:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:23.116 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 1 Processor -- start 2024-09-11T05:20:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 1 -- start start 2024-09-11T05:20:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient: init 2024-09-11T05:20:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-11T05:20:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18b80a8350 addr v1:172.21.15.138:6789/0 2024-09-11T05:20:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f18b80a6250 0x7f18b813db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient(hunting): picked mon.noname-c con 0x7f18b80a6250 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a6660 0x7f18b8140090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient(hunting): picked mon.noname-b con 0x7f18b80a6660 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b809dc70 con 0x7f18b80a8350 2024-09-11T05:20:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f18b8070660 con 0x7f18b80a6250 2024-09-11T05:20:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f18b809d100 con 0x7f18b80a6660 2024-09-11T05:20:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18c048e640 10 monclient(hunting): authenticate will time out at 1706.555392s 2024-09-11T05:20:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a6660 0x7f18b8140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18be22a640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f18b80a8350 0x7f18b80a5b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:32828/0 (socket says 172.21.15.138:32828) 2024-09-11T05:20:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18be22a640 1 -- 172.21.15.138:0/920786966 learned_addr learned my addr 172.21.15.138:0/920786966 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:20:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient(hunting): get_auth_request con 0x7f18b80a6660 auth_method 0 2024-09-11T05:20:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bda29640 1 --2- 172.21.15.138:0/920786966 >> v2:172.21.15.190:3300/0 conn(0x7f18b80a6250 0x7f18b813db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bda29640 10 monclient(hunting): get_auth_request con 0x7f18b80a6250 auth_method 0 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bda29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bda29640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bda29640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient(hunting): handle_auth_done global_id 84347 payload 274 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 1 -- 172.21.15.138:0/920786966 >> v2:172.21.15.190:3300/0 conn(0x7f18b80a6250 msgr2=0x7f18b813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 1 --2- 172.21.15.138:0/920786966 >> v2:172.21.15.190:3300/0 conn(0x7f18b80a6250 0x7f18b813db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 1 -- 172.21.15.138:0/920786966 >> v1:172.21.15.138:6789/0 conn(0x7f18b80a8350 legacy=0x7f18b80a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient: _finish_hunting 0 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 1 monclient: found mon.noname-b 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.124 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.114+0000 7f18bea2b640 1 -- 172.21.15.138:0/920786966 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b809cb40 con 0x7f18b80a6660 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 1 -- 172.21.15.138:0/920786966 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a6660 msgr2=0x7f18b8140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 1 --2- 172.21.15.138:0/920786966 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a6660 0x7f18b8140090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f18a80097c0 tx=0x7f18a801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e471246dee7674c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 1 -- 172.21.15.138:0/920786966 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a80b0070 con 0x7f18b80a6660 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: dump: 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.133 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: _finish_auth 0 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: _check_auth_tickets 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 5 monclient: authenticate success, global_id 84347 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 1 -- 172.21.15.138:0/920786966 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a80ab600 con 0x7f18b80a6660 2024-09-11T05:20:23.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 1 -- 172.21.15.138:0/920786966 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a80ab9d0 con 0x7f18b80a6660 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bd228640 10 monclient: dump: 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 4 set_mon_vals no callback set 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient: shutdown 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- 172.21.15.138:0/920786966 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a6660 msgr2=0x7f18b8140090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 --2- 172.21.15.138:0/920786966 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a6660 0x7f18b8140090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f18a80097c0 tx=0x7f18a801c360 comp rx=0 tx=0).stop 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- 172.21.15.138:0/920786966 shutdown_connections 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 --2- 172.21.15.138:0/920786966 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a6660 0x7f18b8140090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 --2- 172.21.15.138:0/920786966 >> v2:172.21.15.190:3300/0 conn(0x7f18b80a6250 0x7f18b813db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- 172.21.15.138:0/920786966 >> 172.21.15.138:0/920786966 conn(0x7f18b80ab9f0 msgr2=0x7f18b80a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- 172.21.15.138:0/920786966 shutdown_connections 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- 172.21.15.138:0/920786966 wait complete. 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient: build_initial_monmap 2024-09-11T05:20:23.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient: monmap: 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 librados: starting msgr at 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 librados: starting objecter 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 Processor -- start 2024-09-11T05:20:23.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- start start 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 librados: setting wanted keys 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 librados: calling monclient init 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient: init 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18b80a6250 0x7f18b81d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient(hunting): picked mon.noname-b con 0x7f18b80a6250 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 0x7f18b8138990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient(hunting): picked mon.noname-a con 0x7f18b80a6660 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 0x7f18b813da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient(hunting): picked mon.noname-c con 0x7f18b80a8350 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f18b8143000 con 0x7f18b80a6660 2024-09-11T05:20:23.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f18b809da10 con 0x7f18b80a6250 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f18b8143180 con 0x7f18b80a8350 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18c048e640 10 monclient(hunting): authenticate will time out at 1706.563728s 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 0x7f18b813da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bda29640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 0x7f18b8138990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 0x7f18b813da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48100/0 (socket says 172.21.15.138:48100) 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 1 -- 172.21.15.138:0/1422088721 learned_addr learned my addr 172.21.15.138:0/1422088721 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 10 monclient(hunting): get_auth_request con 0x7f18b80a8350 auth_method 0 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bda29640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 0x7f18b8138990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:34384/0 (socket says 172.21.15.138:34384) 2024-09-11T05:20:23.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bda29640 10 monclient(hunting): get_auth_request con 0x7f18b80a6660 auth_method 0 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bda29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bda29640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bda29640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bea2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.122+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18b80a6250 0x7f18b81d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 10 monclient(hunting): handle_auth_done global_id 84350 payload 824 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18b80a6250 msgr2=0x7f18b81d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18b80a6250 0x7f18b81d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 msgr2=0x7f18b8138990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 0x7f18b8138990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 10 monclient: _finish_hunting 0 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 monclient: found mon.noname-c 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b813fd10 con 0x7f18b80a8350 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient: get_auth_request con 0x7f18b80a6250 auth_method 0 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18b80a6250 0x7f18b81d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 0x7f18b8138990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 msgr2=0x7f18b813da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 0x7f18b813da20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f18a801cc00 tx=0x7f18a8019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 client.?.objecter ms_handle_connect 0x7f18b80a8350 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 msgr2=0x7f18b813da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 msgr2=0x7f18b813da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 0x7f18b813da20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f18a801cc00 tx=0x7f18a8019100 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bea2b640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 0x7f18b813da20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f18a801cc00 tx=0x7f18a8019100 comp rx=0 tx=0).stop 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 msgr2=0x7f18b813da20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18b80a8350 0x7f18b813da20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18a4004e40 0x7f18a4007290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f18a4004e40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:23.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 0x7f18a4009dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f18b80a6660 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18b80a6250 0x7f18a400c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f18b80a6250 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f18a400d1d0 con 0x7f18b80a6660 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f18a400d350 con 0x7f18b80a6250 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f18a400d550 con 0x7f18a4004e40 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18b6ffd640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18a4004e40 0x7f18a4007290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient(hunting): get_auth_request con 0x7f18a4004e40 auth_method 0 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 0x7f18a4009dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 10 monclient(hunting): get_auth_request con 0x7f18b80a6660 auth_method 0 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient(hunting): handle_auth_done global_id 84350 payload 293 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18b80a6250 msgr2=0x7f18a400c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18b80a6250 0x7f18a400c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 msgr2=0x7f18a4009dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b80a6660 0x7f18a4009dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient: _finish_hunting 0 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 monclient: found mon.noname-c 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.126+0000 7f18be22a640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a400d810 con 0x7f18a4004e40 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18be22a640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18a4004e40 msgr2=0x7f18a4007290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 client.?.objecter ms_handle_connect 0x7f18a4004e40 2024-09-11T05:20:23.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18be22a640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18a4004e40 0x7f18a4007290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005c00 tx=0x7f18a80a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0005800 con 0x7f18a4004e40 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: dump: 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: _finish_auth 0 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 5 monclient: authenticate success, global_id 84350 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 10 monclient: _renew_subs 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b8121bc0 con 0x7f18a4004e40 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b0013670 con 0x7f18a4004e40 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0013a20 con 0x7f18a4004e40 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 monclient: dump: 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.147 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 10 client.84350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bca27640 4 set_mon_vals no callback set 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 10 monclient: _renew_subs 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b8123320 con 0x7f18a4004e40 2024-09-11T05:20:23.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 1 librados: init done 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b81d02e0 0x7f18b8138990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f18b80a5b40 con 0x7f18b81d02e0 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b81d02e0 0x7f18b8138990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 10 monclient: get_auth_request con 0x7f18b81d02e0 auth_method 0 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f18b0013bc0 con 0x7f18a4004e40 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> v2:172.21.15.190:6800/1562782189 conn(0x7f18b80a8350 0x7f18a40475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 10 monclient(hunting): handle_auth_done global_id 64662 payload 824 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bda29640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18b80008e0 con 0x7f18b81d02e0 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7f18b004f140 con 0x7f18a4004e40 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 10 client.84350.objecter ms_dispatch 0x7f18b8000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 3 client.84350.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 3 client.84350.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18b6ffd640 20 client.84350.objecter dump_active .. 0 homeless 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bea2b640 1 --2- 172.21.15.138:0/1422088721 >> v2:172.21.15.190:6800/1562782189 conn(0x7f18b80a8350 0x7f18a40475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.130+0000 7f18bea2b640 10 monclient: get_auth_request con 0x7f18b80a8350 auth_method 0 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18bda29640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b81d02e0 msgr2=0x7f18b8138990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18bda29640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b81d02e0 0x7f18b8138990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f18ac013c50 tx=0x7f18ac011690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 10 client.84350.objecter ms_handle_connect 0x7f18b81d02e0 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 10 client.84350.objecter resend_mon_ops 2024-09-11T05:20:23.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac004240 con 0x7f18b81d02e0 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 10 monclient: dump: 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f18ac004240 con 0x7f18b81d02e0 2024-09-11T05:20:23.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:23.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:23.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b81d02e0 msgr2=0x7f18b8138990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b81d02e0 0x7f18b8138990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f18ac013c50 tx=0x7f18ac011690 comp rx=0 tx=0).stop 2024-09-11T05:20:23.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b81d02e0 msgr2=0x7f18b8138990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:23.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b81d02e0 0x7f18b8138990 secure :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f18ac013c50 tx=0x7f18ac011690 comp rx=0 tx=0).stop 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18bea2b640 1 --2- 172.21.15.138:0/1422088721 >> v2:172.21.15.190:6800/1562782189 conn(0x7f18b80a8350 0x7f18a40475c0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009580 tx=0x7f18a80b6040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.134+0000 7f18b6ffd640 10 client.84350.objecter ms_handle_connect 0x7f18b80a8350 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18c048e640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18c048e640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b8138ed0 0x7f18b813da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18c048e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f18b8121790 con 0x7f18b8138ed0 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b8138ed0 0x7f18b813da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 10 monclient: get_auth_request con 0x7f18b8138ed0 auth_method 0 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 10 monclient(hunting): handle_auth_done global_id 64665 payload 824 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 1 -- 172.21.15.138:0/1422088721 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f18b8122de0 con 0x7f18b8138ed0 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b8138ed0 msgr2=0x7f18b813da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18b6ffd640 10 client.84350.objecter ms_handle_connect 0x7f18b8138ed0 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18b6ffd640 10 client.84350.objecter resend_mon_ops 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18bda29640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b8138ed0 0x7f18b813da20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f18ac013c50 tx=0x7f18ac004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac022770 con 0x7f18b8138ed0 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.142+0000 7f18b6ffd640 10 monclient: dump: 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.155 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f18ac000e50 con 0x7f18b8138ed0 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b8138ed0 msgr2=0x7f18b813da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b8138ed0 0x7f18b813da20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f18ac013c50 tx=0x7f18ac004740 comp rx=0 tx=0).stop 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18b6ffd640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b8138ed0 msgr2=0x7f18b813da20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:23.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18b6ffd640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f18b8138ed0 0x7f18b813da20 secure :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f18ac013c50 tx=0x7f18ac004740 comp rx=0 tx=0).stop 2024-09-11T05:20:23.157 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:23.157 INFO:teuthology.orchestra.run.smithi138.stdout: "ret": 0, 2024-09-11T05:20:23.157 INFO:teuthology.orchestra.run.smithi138.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T05:20:23.157 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 10 librados: watch_flush enter 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 10 librados: watch_flush exit 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 20 client.84350.objecter shutdown clearing up homeless session... 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 10 client.84350.objecter successfully canceled tick 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 >> v2:172.21.15.190:6800/1562782189 conn(0x7f18b80a8350 msgr2=0x7f18a40475c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 --2- 172.21.15.138:0/1422088721 >> v2:172.21.15.190:6800/1562782189 conn(0x7f18b80a8350 0x7f18a40475c0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009580 tx=0x7f18a80b6040 comp rx=0 tx=0).stop 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 10 monclient: shutdown 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:23.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18a4004e40 msgr2=0x7f18a4007290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18bea2b640 1 -- 172.21.15.138:0/1422088721 reap_dead start 2024-09-11T05:20:23.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18a4004e40 0x7f18a4007290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f18b0005c00 tx=0x7f18a80a6000 comp rx=0 tx=0).stop 2024-09-11T05:20:23.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 shutdown_connections 2024-09-11T05:20:23.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 --2- 172.21.15.138:0/1422088721 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f18a4004e40 0x7f18a4007290 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 >> 172.21.15.138:0/1422088721 conn(0x7f18b80ab9f0 msgr2=0x7f18b80a9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:23.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 shutdown_connections 2024-09-11T05:20:23.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 -- 172.21.15.138:0/1422088721 wait complete. 2024-09-11T05:20:23.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.146+0000 7f18c048e640 1 librados: shutdown 2024-09-11T05:20:23.160 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T05:20:23.160 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T05:20:23.160 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:20:23.218 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T05:20:23.218 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:20:23.218 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:20:23.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient: get_monmap_and_config 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient: build_initial_monmap 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient: monmap: 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:23.379154+0000 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:23.379154+0000 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:23.374 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:23.375 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:23.375 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 1 Processor -- start 2024-09-11T05:20:23.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 1 -- start start 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient: init 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f47e009dc20 0x7f47e00a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient(hunting): picked mon.noname-c con 0x7f47e009dc20 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e00a2670 0x7f47e00a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient(hunting): picked mon.noname-b con 0x7f47e00a2670 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47e00a0a90 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e009efa0 con 0x7f47e00a0a90 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f47e00a37b0 con 0x7f47e009dc20 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f47e009ecc0 con 0x7f47e00a2670 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47e67a2640 10 monclient(hunting): authenticate will time out at 1706.815257s 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e00a2670 0x7f47e00a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e00a2670 0x7f47e00a2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48148/0 (socket says 172.21.15.138:48148) 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 1 -- 172.21.15.138:0/3292325140 learned_addr learned my addr 172.21.15.138:0/3292325140 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient(hunting): get_auth_request con 0x7f47e00a2670 auth_method 0 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient(hunting): handle_auth_done global_id 84353 payload 274 2024-09-11T05:20:23.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47dffff640 1 --2- 172.21.15.138:0/3292325140 >> v2:172.21.15.190:3300/0 conn(0x7f47e009dc20 0x7f47e00a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 1 -- 172.21.15.138:0/3292325140 >> v2:172.21.15.190:3300/0 conn(0x7f47e009dc20 msgr2=0x7f47e00a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:23.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 1 --2- 172.21.15.138:0/3292325140 >> v2:172.21.15.190:3300/0 conn(0x7f47e009dc20 0x7f47e00a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 1 -- 172.21.15.138:0/3292325140 >> v1:172.21.15.190:6789/0 conn(0x7f47e00a0a90 legacy=0x7f47e01348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:23.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:20:23.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 1 monclient: found mon.noname-b 2024-09-11T05:20:23.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:23.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.374+0000 7f47df7fe640 1 -- 172.21.15.138:0/3292325140 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e006e340 con 0x7f47e00a2670 2024-09-11T05:20:23.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47df7fe640 1 -- 172.21.15.138:0/3292325140 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e00a2670 msgr2=0x7f47e00a2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:23.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47df7fe640 1 --2- 172.21.15.138:0/3292325140 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e00a2670 0x7f47e00a2a50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f47d400bb10 tx=0x7f47d401e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7dff2989fe3e990 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 1 -- 172.21.15.138:0/3292325140 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40b4070 con 0x7f47e00a2670 2024-09-11T05:20:23.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.381 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: dump: 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: _finish_auth 0 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: _check_auth_tickets 2024-09-11T05:20:23.382 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 1 -- 172.21.15.138:0/3292325140 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d40a7a60 con 0x7f47e00a2670 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 1 -- 172.21.15.138:0/3292325140 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40a7e30 con 0x7f47e00a2670 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 5 monclient: authenticate success, global_id 84353 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 4 set_mon_vals no callback set 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47deffd640 10 monclient: dump: 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.383 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient: shutdown 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- 172.21.15.138:0/3292325140 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e00a2670 msgr2=0x7f47e00a2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 --2- 172.21.15.138:0/3292325140 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e00a2670 0x7f47e00a2a50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f47d400bb10 tx=0x7f47d401e910 comp rx=0 tx=0).stop 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- 172.21.15.138:0/3292325140 shutdown_connections 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 --2- 172.21.15.138:0/3292325140 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e00a2670 0x7f47e00a2a50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 --2- 172.21.15.138:0/3292325140 >> v2:172.21.15.190:3300/0 conn(0x7f47e009dc20 0x7f47e00a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- 172.21.15.138:0/3292325140 >> 172.21.15.138:0/3292325140 conn(0x7f47e0063640 msgr2=0x7f47e0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:23.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- 172.21.15.138:0/3292325140 shutdown_connections 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- 172.21.15.138:0/3292325140 wait complete. 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient: build_initial_monmap 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient: monmap: 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 librados: starting msgr at 2024-09-11T05:20:23.385 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 librados: starting objecter 2024-09-11T05:20:23.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 Processor -- start 2024-09-11T05:20:23.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- start start 2024-09-11T05:20:23.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 librados: setting wanted keys 2024-09-11T05:20:23.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 librados: calling monclient init 2024-09-11T05:20:23.386 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient: init 2024-09-11T05:20:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:20:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e009dc20 0x7f47e01ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient(hunting): picked mon.noname-c con 0x7f47e009dc20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47e00a0a90 0x7f47e01cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient(hunting): picked mon.noname-b con 0x7f47e00a0a90 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f47e00a2670 0x7f47e013a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.390 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient(hunting): picked mon.noname-a con 0x7f47e00a2670 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f47e0135f90 con 0x7f47e00a2670 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f47e0135e10 con 0x7f47e00a0a90 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f47e0136110 con 0x7f47e009dc20 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e67a2640 10 monclient(hunting): authenticate will time out at 1706.822510s 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e4d3f640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f47e00a2670 msgr2=0x7f47e013a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47e4d3f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f47e00a2670 0x7f47e013a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e009dc20 0x7f47e01ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e009dc20 0x7f47e01ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48158/0 (socket says 172.21.15.138:48158) 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 -- 172.21.15.138:0/1817690386 learned_addr learned my addr 172.21.15.138:0/1817690386 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient(hunting): get_auth_request con 0x7f47e009dc20 auth_method 0 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47df7fe640 1 --2- 172.21.15.138:0/1817690386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47e00a0a90 0x7f47e01cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47df7fe640 10 monclient(hunting): get_auth_request con 0x7f47e00a0a90 auth_method 0 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient(hunting): handle_auth_done global_id 84356 payload 824 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 -- 172.21.15.138:0/1817690386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47e00a0a90 msgr2=0x7f47e01cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 --2- 172.21.15.138:0/1817690386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47e00a0a90 0x7f47e01cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 -- 172.21.15.138:0/1817690386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f47e00a2670 msgr2=0x7f47e013a7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:20:23.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 --2- 172.21.15.138:0/1817690386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f47e00a2670 0x7f47e013a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient: _finish_hunting 0 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 monclient: found mon.noname-c 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.382+0000 7f47dffff640 1 -- 172.21.15.138:0/1817690386 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e011e070 con 0x7f47e009dc20 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dffff640 1 -- 172.21.15.138:0/1817690386 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e009dc20 msgr2=0x7f47e01ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 client.?.objecter ms_handle_connect 0x7f47e009dc20 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dffff640 1 --2- 172.21.15.138:0/1817690386 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e009dc20 0x7f47e01ccd10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f47d000c7f0 tx=0x7f47d000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 1 -- 172.21.15.138:0/1817690386 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d0015070 con 0x7f47e009dc20 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:23.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: dump: 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: _finish_auth 0 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: _check_auth_tickets 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:23.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 5 monclient: authenticate success, global_id 84356 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 1 -- 172.21.15.138:0/1817690386 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d0004240 con 0x7f47e009dc20 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 10 monclient: _renew_subs 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e00a5e30 con 0x7f47e009dc20 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 1 -- 172.21.15.138:0/1817690386 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d00077d0 con 0x7f47e009dc20 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 monclient: dump: 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:23.395 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 10 client.84356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 10 monclient: _renew_subs 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47de7fc640 4 set_mon_vals no callback set 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e011dd90 con 0x7f47e009dc20 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 1 librados: init done 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:23.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e00a1920 con 0x7f47e009dc20 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 1 -- 172.21.15.138:0/1817690386 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f47d00079b0 con 0x7f47e009dc20 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 1 --2- 172.21.15.138:0/1817690386 >> v2:172.21.15.190:6800/1562782189 conn(0x7f47cc03ce50 0x7f47cc03f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 1 -- 172.21.15.138:0/1817690386 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7f47d004fc00 con 0x7f47e009dc20 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 10 client.84356.objecter ms_dispatch 0x7f47e0000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 3 client.84356.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 3 client.84356.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47dcff9640 20 client.84356.objecter dump_active .. 0 homeless 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47df7fe640 1 --2- 172.21.15.138:0/1817690386 >> v2:172.21.15.190:6800/1562782189 conn(0x7f47cc03ce50 0x7f47cc03f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.386+0000 7f47df7fe640 10 monclient: get_auth_request con 0x7f47cc03ce50 auth_method 0 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.390+0000 7f47df7fe640 1 --2- 172.21.15.138:0/1817690386 >> v2:172.21.15.190:6800/1562782189 conn(0x7f47cc03ce50 0x7f47cc03f310 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f47e006a140 tx=0x7f47d40a9070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:23.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.390+0000 7f47dcff9640 10 client.84356.objecter ms_handle_connect 0x7f47cc03ce50 2024-09-11T05:20:23.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.390+0000 7f47dcff9640 1 -- 172.21.15.138:0/1817690386 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f47d0007cd0 con 0x7f47e009dc20 2024-09-11T05:20:23.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.390+0000 7f47dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:23.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.390+0000 7f47dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:23.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.622+0000 7f47e67a2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:20:23.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.622+0000 7f47e67a2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:23.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.622+0000 7f47e67a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:23.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.622+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f47e006ccd0 con 0x7f47e009dc20 2024-09-11T05:20:23.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47dcff9640 1 -- 172.21.15.138:0/1817690386 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f47d0003b40 con 0x7f47e009dc20 2024-09-11T05:20:23.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:23.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:23.625 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:20:23.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 10 librados: watch_flush enter 2024-09-11T05:20:23.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 10 librados: watch_flush exit 2024-09-11T05:20:23.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 20 client.84356.objecter shutdown clearing up homeless session... 2024-09-11T05:20:23.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 10 client.84356.objecter successfully canceled tick 2024-09-11T05:20:23.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 >> v2:172.21.15.190:6800/1562782189 conn(0x7f47cc03ce50 msgr2=0x7f47cc03f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 --2- 172.21.15.138:0/1817690386 >> v2:172.21.15.190:6800/1562782189 conn(0x7f47cc03ce50 0x7f47cc03f310 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f47e006a140 tx=0x7f47d40a9070 comp rx=0 tx=0).stop 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 10 monclient: shutdown 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e009dc20 msgr2=0x7f47e01ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 --2- 172.21.15.138:0/1817690386 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e009dc20 0x7f47e01ccd10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f47d000c7f0 tx=0x7f47d000ccb0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 shutdown_connections 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 --2- 172.21.15.138:0/1817690386 >> v2:172.21.15.190:6800/1562782189 conn(0x7f47cc03ce50 0x7f47cc03f310 unknown :-1 s=CLOSED pgs=3463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 --2- 172.21.15.138:0/1817690386 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f47e00a2670 0x7f47e013a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 --2- 172.21.15.138:0/1817690386 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f47e00a0a90 0x7f47e01cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 --2- 172.21.15.138:0/1817690386 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f47e009dc20 0x7f47e01ccd10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 >> 172.21.15.138:0/1817690386 conn(0x7f47e0063640 msgr2=0x7f47e0071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 shutdown_connections 2024-09-11T05:20:23.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 -- 172.21.15.138:0/1817690386 wait complete. 2024-09-11T05:20:23.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:23.626+0000 7f47e67a2640 1 librados: shutdown 2024-09-11T05:20:26.645 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:20:26.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient: get_monmap_and_config 2024-09-11T05:20:26.773 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient: build_initial_monmap 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient: monmap: 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:26.779226+0000 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:26.779226+0000 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:26.774 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:26.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 1 Processor -- start 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 1 -- start start 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient: init 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f619c0a5b40 0x7f619c0a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient(hunting): picked mon.noname-a con 0x7f619c0a5b40 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f619c0a7130 0x7f619c0a6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient(hunting): picked mon.noname-c con 0x7f619c0a7130 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f619c09d840 addr v1:172.21.15.138:6789/0 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f619c09d490 con 0x7f619c09d840 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:26.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f619c09e110 con 0x7f619c0a5b40 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f619c0a69a0 con 0x7f619c0a7130 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f61a09c2640 10 monclient(hunting): authenticate will time out at 1710.215368s 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f619a59c640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7f619c0a5b40 msgr2=0x7f619c0a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f619a59c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f619c0a5b40 0x7f619c0a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f619ad9d640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7f619c09d840 legacy=0x7f619c138900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f6199d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f619c0a7130 0x7f619c0a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f6199d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f619c0a7130 0x7f619c0a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.138:36158/0 (socket says 172.21.15.138:36158) 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f6199d9b640 1 -- 172.21.15.138:0/3934297617 learned_addr learned my addr 172.21.15.138:0/3934297617 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f6199d9b640 10 monclient(hunting): get_auth_request con 0x7f619c0a7130 auth_method 0 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f6199d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:26.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f6199d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.774+0000 7f6199d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 10 monclient(hunting): handle_auth_done global_id 104131 payload 274 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 1 -- 172.21.15.138:0/3934297617 >> v2:172.21.15.138:3300/0 conn(0x7f619c0a5b40 msgr2=0x7f619c0a5f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 1 --2- 172.21.15.138:0/3934297617 >> v2:172.21.15.138:3300/0 conn(0x7f619c0a5b40 0x7f619c0a5f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 1 -- 172.21.15.138:0/3934297617 >> v1:172.21.15.138:6789/0 conn(0x7f619c09d840 legacy=0x7f619c138900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 1 monclient: found mon.noname-c 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:20:26.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.778+0000 7f6199d9b640 1 -- 172.21.15.138:0/3934297617 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c070580 con 0x7f619c0a7130 2024-09-11T05:20:26.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 1 -- 172.21.15.138:0/3934297617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 msgr2=0x7f619c0a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:26.780 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 1 --2- 172.21.15.138:0/3934297617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 0x7f619c0a6460 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f619c0a22e0 tx=0x7f619001c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc9fa2b0b66868d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:26.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 1 -- 172.21.15.138:0/3934297617 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61900b0070 con 0x7f619c0a7130 2024-09-11T05:20:26.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:26.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: dump: 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:26.783 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: _finish_auth 0 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: _check_auth_tickets 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 1 -- 172.21.15.138:0/3934297617 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61900ab600 con 0x7f619c0a7130 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 5 monclient: authenticate success, global_id 104131 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 4 set_mon_vals no callback set 2024-09-11T05:20:26.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 1 -- 172.21.15.138:0/3934297617 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61900ab9d0 con 0x7f619c0a7130 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619959a640 10 monclient: dump: 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:26.785 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient: shutdown 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- 172.21.15.138:0/3934297617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 msgr2=0x7f619c0a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 --2- 172.21.15.138:0/3934297617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 0x7f619c0a6460 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f619c0a22e0 tx=0x7f619001c400 comp rx=0 tx=0).stop 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- 172.21.15.138:0/3934297617 shutdown_connections 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 --2- 172.21.15.138:0/3934297617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 0x7f619c0a6460 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 --2- 172.21.15.138:0/3934297617 >> v2:172.21.15.138:3300/0 conn(0x7f619c0a5b40 0x7f619c0a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- 172.21.15.138:0/3934297617 >> 172.21.15.138:0/3934297617 conn(0x7f619c0aba10 msgr2=0x7f619c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- 172.21.15.138:0/3934297617 shutdown_connections 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- 172.21.15.138:0/3934297617 wait complete. 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient: build_initial_monmap 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient: monmap: 2024-09-11T05:20:26.786 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 librados: starting msgr at 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 librados: starting objecter 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 Processor -- start 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- start start 2024-09-11T05:20:26.787 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 librados: setting wanted keys 2024-09-11T05:20:26.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 librados: calling monclient init 2024-09-11T05:20:26.788 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient: init 2024-09-11T05:20:26.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:26.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:20:26.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f619c09d840 0x7f619c1d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient(hunting): picked mon.noname-a con 0x7f619c09d840 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f619c0a5b40 0x7f619c1d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient(hunting): picked mon.noname-c con 0x7f619c0a5b40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 0x7f619c13ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient(hunting): picked mon.noname-b con 0x7f619c0a7130 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f619c142f60 con 0x7f619c09d840 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f619c142de0 con 0x7f619c0a7130 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f619c1430e0 con 0x7f619c0a5b40 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f61a09c2640 10 monclient(hunting): authenticate will time out at 1710.223737s 2024-09-11T05:20:26.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619a59c640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f619c09d840 msgr2=0x7f619c1d06e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f619a59c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f619c09d840 0x7f619c1d06e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f619c0a5b40 0x7f619c1d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f619c0a5b40 0x7f619c1d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48164/0 (socket says 172.21.15.138:48164) 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 1 -- 172.21.15.138:0/2237881572 learned_addr learned my addr 172.21.15.138:0/2237881572 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 10 monclient(hunting): get_auth_request con 0x7f619c0a5b40 auth_method 0 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.782+0000 7f6199d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f619ad9d640 1 --2- 172.21.15.138:0/2237881572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 0x7f619c13ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f619ad9d640 10 monclient(hunting): get_auth_request con 0x7f619c0a7130 auth_method 0 2024-09-11T05:20:26.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f619ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f619ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f619ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 10 monclient(hunting): handle_auth_done global_id 84359 payload 824 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 1 -- 172.21.15.138:0/2237881572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 msgr2=0x7f619c13ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 1 --2- 172.21.15.138:0/2237881572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 0x7f619c13ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 1 -- 172.21.15.138:0/2237881572 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f619c09d840 msgr2=0x7f619c1d06e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 1 --2- 172.21.15.138:0/2237881572 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f619c09d840 0x7f619c1d06e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 1 monclient: found mon.noname-c 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 1 -- 172.21.15.138:0/2237881572 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c0a4ac0 con 0x7f619c0a5b40 2024-09-11T05:20:26.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 1 -- 172.21.15.138:0/2237881572 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f619c0a5b40 msgr2=0x7f619c1d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6199d9b640 1 --2- 172.21.15.138:0/2237881572 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f619c0a5b40 0x7f619c1d0e10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f619001a030 tx=0x7f61900191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 client.?.objecter ms_handle_connect 0x7f619c0a5b40 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 1 -- 172.21.15.138:0/2237881572 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61900b9020 con 0x7f619c0a5b40 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: dump: 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:26.796 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: _finish_auth 0 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: _check_auth_tickets 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 5 monclient: authenticate success, global_id 84359 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 10 monclient: _renew_subs 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619c121ee0 con 0x7f619c0a5b40 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 1 -- 172.21.15.138:0/2237881572 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61900b1040 con 0x7f619c0a5b40 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 1 -- 172.21.15.138:0/2237881572 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61900b0050 con 0x7f619c0a5b40 2024-09-11T05:20:26.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 10 monclient: dump: 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f6198d99640 4 set_mon_vals no callback set 2024-09-11T05:20:26.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 10 client.84359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 10 monclient: _renew_subs 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f619c121820 con 0x7f619c0a5b40 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 1 librados: init done 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f619c138900 con 0x7f619c0a5b40 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.786+0000 7f618affd640 1 -- 172.21.15.138:0/2237881572 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f61900b5a50 con 0x7f619c0a5b40 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f618affd640 1 --2- 172.21.15.138:0/2237881572 >> v2:172.21.15.190:6800/1562782189 conn(0x7f617c03ce50 0x7f617c03f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f618affd640 1 -- 172.21.15.138:0/2237881572 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7f61900e9f60 con 0x7f619c0a5b40 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f618affd640 10 client.84359.objecter ms_dispatch 0x7f619c000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:26.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f618affd640 3 client.84359.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f618affd640 3 client.84359.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f618affd640 20 client.84359.objecter dump_active .. 0 homeless 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f619a59c640 1 --2- 172.21.15.138:0/2237881572 >> v2:172.21.15.190:6800/1562782189 conn(0x7f617c03ce50 0x7f617c03f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f619a59c640 10 monclient: get_auth_request con 0x7f617c03ce50 auth_method 0 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f619a59c640 1 --2- 172.21.15.138:0/2237881572 >> v2:172.21.15.190:6800/1562782189 conn(0x7f617c03ce50 0x7f617c03f310 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f618c006fd0 tx=0x7f618c006d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.790+0000 7f618affd640 10 client.84359.objecter ms_handle_connect 0x7f617c03ce50 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.794+0000 7f618affd640 1 -- 172.21.15.138:0/2237881572 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f61900c3030 con 0x7f619c0a5b40 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.794+0000 7f618affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:26.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:26.794+0000 7f618affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:27.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.026+0000 7f61a09c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:20:27.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.026+0000 7f61a09c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:27.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.026+0000 7f61a09c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:27.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.026+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f619c1227b0 con 0x7f619c0a5b40 2024-09-11T05:20:27.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f618affd640 1 -- 172.21.15.138:0/2237881572 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f619c121ee0 con 0x7f619c0a5b40 2024-09-11T05:20:27.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f618affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:27.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f618affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:27.028 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:20:27.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 10 librados: watch_flush enter 2024-09-11T05:20:27.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 10 librados: watch_flush exit 2024-09-11T05:20:27.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 20 client.84359.objecter shutdown clearing up homeless session... 2024-09-11T05:20:27.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 10 client.84359.objecter successfully canceled tick 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 >> v2:172.21.15.190:6800/1562782189 conn(0x7f617c03ce50 msgr2=0x7f617c03f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 --2- 172.21.15.138:0/2237881572 >> v2:172.21.15.190:6800/1562782189 conn(0x7f617c03ce50 0x7f617c03f310 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f618c006fd0 tx=0x7f618c006d40 comp rx=0 tx=0).stop 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 10 monclient: shutdown 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f619c0a5b40 msgr2=0x7f619c1d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 --2- 172.21.15.138:0/2237881572 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f619c0a5b40 0x7f619c1d0e10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f619001a030 tx=0x7f61900191a0 comp rx=0 tx=0).stop 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 shutdown_connections 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 --2- 172.21.15.138:0/2237881572 >> v2:172.21.15.190:6800/1562782189 conn(0x7f617c03ce50 0x7f617c03f310 unknown :-1 s=CLOSED pgs=3464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 --2- 172.21.15.138:0/2237881572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f619c0a7130 0x7f619c13ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 --2- 172.21.15.138:0/2237881572 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f619c0a5b40 0x7f619c1d0e10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 --2- 172.21.15.138:0/2237881572 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f619c09d840 0x7f619c1d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 >> 172.21.15.138:0/2237881572 conn(0x7f619c0aba10 msgr2=0x7f619c0a43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 shutdown_connections 2024-09-11T05:20:27.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 -- 172.21.15.138:0/2237881572 wait complete. 2024-09-11T05:20:27.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:27.030+0000 7f61a09c2640 1 librados: shutdown 2024-09-11T05:20:30.047 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient: get_monmap_and_config 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient: build_initial_monmap 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient: monmap: 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:30.178800+0000 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:30.178800+0000 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:30.173 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:30.174 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:30.174 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:30.174 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:30.174 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:30.174 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:30.174 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 1 Processor -- start 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 1 -- start start 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient: init 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f08140a5c50 0x7f08140a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient(hunting): picked mon.noname-b con 0x7f08140a5c50 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08140a6570 addr v1:172.21.15.138:6789/0 2024-09-11T05:20:30.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f081413e150 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08140705b0 con 0x7f08140a6570 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08140a2990 con 0x7f081413e150 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f08140a32c0 con 0x7f08140a5c50 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819622640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7f08140a6570 legacy=0x7f081413ca30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f081c087640 10 monclient(hunting): authenticate will time out at 1713.615001s 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f08140a5c50 0x7f08140a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f08140a5c50 0x7f08140a6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:33158/0 (socket says 172.21.15.138:33158) 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 1 -- 172.21.15.138:0/987440205 learned_addr learned my addr 172.21.15.138:0/987440205 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient(hunting): get_auth_request con 0x7f08140a5c50 auth_method 0 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:30.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient(hunting): handle_auth_done global_id 84362 payload 274 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 1 -- 172.21.15.138:0/987440205 >> v1:172.21.15.190:6789/0 conn(0x7f081413e150 legacy=0x7f0814140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 1 -- 172.21.15.138:0/987440205 >> v1:172.21.15.138:6789/0 conn(0x7f08140a6570 legacy=0x7f081413ca30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient: _finish_hunting 0 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 1 monclient: found mon.noname-b 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:20:30.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.174+0000 7f0819e23640 1 -- 172.21.15.138:0/987440205 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08140a2c70 con 0x7f08140a5c50 2024-09-11T05:20:30.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0819e23640 1 -- 172.21.15.138:0/987440205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f08140a5c50 msgr2=0x7f08140a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:30.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0819e23640 1 --2- 172.21.15.138:0/987440205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f08140a5c50 0x7f08140a6030 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f08080098a0 tx=0x7f080801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b2ba5933058b4a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:30.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 1 -- 172.21.15.138:0/987440205 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08080b0070 con 0x7f08140a5c50 2024-09-11T05:20:30.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:30.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:20:30.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: dump: 2024-09-11T05:20:30.180 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: _finish_auth 0 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: _check_auth_tickets 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:30.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 1 -- 172.21.15.138:0/987440205 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08080ab400 con 0x7f08140a5c50 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f081c087640 5 monclient: authenticate success, global_id 84362 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f081c087640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f081c087640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 1 -- 172.21.15.138:0/987440205 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08080ab7d0 con 0x7f08140a5c50 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f081c087640 4 set_mon_vals no callback set 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.178+0000 7f0818e21640 10 monclient: dump: 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:30.182 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient: shutdown 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- 172.21.15.138:0/987440205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f08140a5c50 msgr2=0x7f08140a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 --2- 172.21.15.138:0/987440205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f08140a5c50 0x7f08140a6030 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f08080098a0 tx=0x7f080801c440 comp rx=0 tx=0).stop 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- 172.21.15.138:0/987440205 shutdown_connections 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 --2- 172.21.15.138:0/987440205 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f08140a5c50 0x7f08140a6030 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- 172.21.15.138:0/987440205 >> 172.21.15.138:0/987440205 conn(0x7f08140a8620 msgr2=0x7f0814063640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- 172.21.15.138:0/987440205 shutdown_connections 2024-09-11T05:20:30.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- 172.21.15.138:0/987440205 wait complete. 2024-09-11T05:20:30.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient: build_initial_monmap 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient: monmap: 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:30.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 librados: starting msgr at 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 librados: starting objecter 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 Processor -- start 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- start start 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 librados: setting wanted keys 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 librados: calling monclient init 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient: init 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08140a5c50 0x7f08141cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient(hunting): picked mon.noname-a con 0x7f08140a5c50 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08140a6570 0x7f08141ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient(hunting): picked mon.noname-b con 0x7f08140a6570 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f081413e150 0x7f081409e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:30.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient(hunting): picked mon.noname-c con 0x7f081413e150 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f08140a5730 con 0x7f08140a5c50 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f08140a55b0 con 0x7f08140a6570 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f0814144480 con 0x7f081413e150 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081c087640 10 monclient(hunting): authenticate will time out at 1713.622157s 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f0819e23640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08140a5c50 msgr2=0x7f08141cc810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f0819e23640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08140a5c50 0x7f08141cc810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f081413e150 0x7f081409e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f081413e150 0x7f081409e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:33160/0 (socket says 172.21.15.138:33160) 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 -- 172.21.15.138:0/3680191542 learned_addr learned my addr 172.21.15.138:0/3680191542 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:30.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient(hunting): get_auth_request con 0x7f081413e150 auth_method 0 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient(hunting): handle_auth_done global_id 84365 payload 824 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f0819622640 1 --2- 172.21.15.138:0/3680191542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08140a6570 0x7f08141ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 -- 172.21.15.138:0/3680191542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08140a6570 msgr2=0x7f08141ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 --2- 172.21.15.138:0/3680191542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08140a6570 0x7f08141ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 -- 172.21.15.138:0/3680191542 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08140a5c50 msgr2=0x7f08141cc810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 --2- 172.21.15.138:0/3680191542 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08140a5c50 0x7f08141cc810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient: _finish_hunting 0 2024-09-11T05:20:30.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 monclient: found mon.noname-c 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.182+0000 7f081a624640 1 -- 172.21.15.138:0/3680191542 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0814138f80 con 0x7f081413e150 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081a624640 1 -- 172.21.15.138:0/3680191542 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f081413e150 msgr2=0x7f081409e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081a624640 1 --2- 172.21.15.138:0/3680191542 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f081413e150 0x7f081409e800 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f080c00c780 tx=0x7f080c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 client.?.objecter ms_handle_connect 0x7f081413e150 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 1 -- 172.21.15.138:0/3680191542 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080c017070 con 0x7f081413e150 2024-09-11T05:20:30.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: dump: 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: _finish_auth 0 2024-09-11T05:20:30.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: _check_auth_tickets 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 5 monclient: authenticate success, global_id 84365 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 1 -- 172.21.15.138:0/3680191542 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f080c00ce30 con 0x7f081413e150 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 1 -- 172.21.15.138:0/3680191542 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080c005290 con 0x7f081413e150 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 10 monclient: _renew_subs 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f0807fff640 4 set_mon_vals no callback set 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0814139260 con 0x7f081413e150 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 monclient: dump: 2024-09-11T05:20:30.194 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 10 client.84365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 10 monclient: _renew_subs 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0814139540 con 0x7f081413e150 2024-09-11T05:20:30.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 1 librados: init done 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 1 -- 172.21.15.138:0/3680191542 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f080c005470 con 0x7f081413e150 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08140a6030 con 0x7f081413e150 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 1 --2- 172.21.15.138:0/3680191542 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07f803cea0 0x7f07f803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 1 -- 172.21.15.138:0/3680191542 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7f080c04f590 con 0x7f081413e150 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 10 client.84365.objecter ms_dispatch 0x7f0814000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 3 client.84365.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 3 client.84365.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f08067fc640 20 client.84365.objecter dump_active .. 0 homeless 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f0819e23640 1 --2- 172.21.15.138:0/3680191542 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07f803cea0 0x7f07f803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:30.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.186+0000 7f0819e23640 10 monclient: get_auth_request con 0x7f07f803cea0 auth_method 0 2024-09-11T05:20:30.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.190+0000 7f08067fc640 10 client.84365.objecter ms_handle_connect 0x7f07f803cea0 2024-09-11T05:20:30.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.190+0000 7f0819e23640 1 --2- 172.21.15.138:0/3680191542 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07f803cea0 0x7f07f803f360 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f08140a8e10 tx=0x7f0808008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:30.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.190+0000 7f08067fc640 1 -- 172.21.15.138:0/3680191542 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f080c005790 con 0x7f081413e150 2024-09-11T05:20:30.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.190+0000 7f08067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:30.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.190+0000 7f08067fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:30.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.414+0000 7f081c087640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:20:30.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.414+0000 7f081c087640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:30.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.414+0000 7f081c087640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:30.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.414+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f081406cc20 con 0x7f081413e150 2024-09-11T05:20:30.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.414+0000 7f08067fc640 1 -- 172.21.15.138:0/3680191542 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f080c04db60 con 0x7f081413e150 2024-09-11T05:20:30.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.414+0000 7f08067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:30.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.414+0000 7f08067fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:30.414 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:20:30.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.414+0000 7f081c087640 10 librados: watch_flush enter 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 10 librados: watch_flush exit 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 20 client.84365.objecter shutdown clearing up homeless session... 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 10 client.84365.objecter successfully canceled tick 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07f803cea0 msgr2=0x7f07f803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 --2- 172.21.15.138:0/3680191542 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07f803cea0 0x7f07f803f360 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f08140a8e10 tx=0x7f0808008040 comp rx=0 tx=0).stop 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 10 monclient: shutdown 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f081413e150 msgr2=0x7f081409e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 --2- 172.21.15.138:0/3680191542 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f081413e150 0x7f081409e800 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f080c00c780 tx=0x7f080c00cc40 comp rx=0 tx=0).stop 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 shutdown_connections 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 --2- 172.21.15.138:0/3680191542 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07f803cea0 0x7f07f803f360 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 --2- 172.21.15.138:0/3680191542 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f081413e150 0x7f081409e800 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 --2- 172.21.15.138:0/3680191542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08140a6570 0x7f08141ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:30.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 --2- 172.21.15.138:0/3680191542 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f08140a5c50 0x7f08141cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:30.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 >> 172.21.15.138:0/3680191542 conn(0x7f08140a8620 msgr2=0x7f0814070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:30.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 shutdown_connections 2024-09-11T05:20:30.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 -- 172.21.15.138:0/3680191542 wait complete. 2024-09-11T05:20:30.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:30.418+0000 7f081c087640 1 librados: shutdown 2024-09-11T05:20:33.433 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:20:33.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient: get_monmap_and_config 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient: build_initial_monmap 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient: monmap: 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:33.563453+0000 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:33.563453+0000 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:33.558 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:33.559 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:33.559 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:33.559 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:33.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 1 Processor -- start 2024-09-11T05:20:33.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 1 -- start start 2024-09-11T05:20:33.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient: init 2024-09-11T05:20:33.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:33.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-11T05:20:33.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea340a4030 addr v1:172.21.15.138:6789/0 2024-09-11T05:20:33.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient(hunting): picked mon.noname-c con 0x7fea340a4b20 addr v2:172.21.15.190:3300/0 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient(hunting): picked mon.noname-b con 0x7fea3413da40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea340a2850 con 0x7fea340a4030 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fea340a30f0 con 0x7fea340a4b20 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fea340a2b30 con 0x7fea3413da40 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea393a9640 10 monclient(hunting): authenticate will time out at 1716.999642s 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea32ffd640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7fea340a4030 legacy=0x7fea340a4410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:33.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:33174/0 (socket says 172.21.15.138:33174) 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 learned_addr learned my addr 172.21.15.138:0/1452978505 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 10 monclient(hunting): get_auth_request con 0x7fea3413da40 auth_method 0 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.558+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 10 monclient(hunting): get_auth_request con 0x7fea340a4b20 auth_method 0 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:33.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:20:33.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:33.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:33.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.562+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:33.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> v1:172.21.15.138:6789/0 conn(0x7fea340a4030 legacy=0x7fea340a4410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:20:33.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:33.761 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 10 monclient(hunting): get_auth_request con 0x7fea3413da40 auth_method 0 2024-09-11T05:20:33.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:33.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:33.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:33.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:20:33.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:33.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:33.762 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.762+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:20:33.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:33.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 10 monclient(hunting): get_auth_request con 0x7fea340a4b20 auth_method 0 2024-09-11T05:20:33.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:33.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:33.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:33.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:20:33.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:33.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:33.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:33.766+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:20:34.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 10 monclient(hunting): get_auth_request con 0x7fea3413da40 auth_method 0 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:20:34.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.166+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> v1:172.21.15.138:6789/0 conn(0x7fea340a4030 legacy=0x7fea340a4410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:20:34.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:34.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 10 monclient(hunting): get_auth_request con 0x7fea340a4b20 auth_method 0 2024-09-11T05:20:34.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:34.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:34.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:34.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:20:34.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:34.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:34.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.170+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:20:34.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> v1:172.21.15.138:6789/0 conn(0x7fea340a4030 legacy=0x7fea340a4410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:20:34.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:34.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 10 monclient(hunting): get_auth_request con 0x7fea3413da40 auth_method 0 2024-09-11T05:20:34.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:34.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:34.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:34.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:20:34.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:34.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:34.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.970+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:20:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 10 monclient(hunting): get_auth_request con 0x7fea340a4b20 auth_method 0 2024-09-11T05:20:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:20:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:20:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:20:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:34.974+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 10 monclient(hunting): tick 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 monclient(hunting): continuing hunt 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea3413fe30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea3413fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 msgr2=0x7fea340a3ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.190:3300/0 conn(0x7fea340a4b20 0x7fea340a3ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:36.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 -- 172.21.15.138:0/1452978505 >> v1:172.21.15.138:6789/0 conn(0x7fea340a4030 legacy=0x7fea340a4410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 10 monclient: _add_conns ranks=[3,1] 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 0x7fea20005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fea200036b0 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea20006160 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 -- 172.21.15.138:0/1452978505 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea340a2e10 con 0x7fea20006160 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 1 -- 172.21.15.138:0/1452978505 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fea340a30f0 con 0x7fea200036b0 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.558+0000 7fea30ff9640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.562+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 msgr2=0x7fea20005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:36.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.562+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 0x7fea20005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:36.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.562+0000 7fea327fc640 1 -- 172.21.15.138:0/1452978505 >> v1:172.21.15.190:6789/0 conn(0x7fea20006160 legacy=0x7fea20008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:36.764 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.766+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 msgr2=0x7fea20005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:36.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:36.766+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 0x7fea20005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:20:37.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.170+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 msgr2=0x7fea20005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:37.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.170+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 0x7fea20005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:20:37.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.262+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183633682 0 0) 0x7fea340a2e10 con 0x7fea20006160 2024-09-11T05:20:37.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.262+0000 7fea31ffb640 10 monclient(hunting): _negotiate 2024-09-11T05:20:37.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.262+0000 7fea31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.262+0000 7fea31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.262+0000 7fea31ffb640 10 monclient(hunting): my global_id is 104137 2024-09-11T05:20:37.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.262+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea1c003530 con 0x7fea20006160 2024-09-11T05:20:37.270 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3346502996 0 0) 0x7fea1c003530 con 0x7fea20006160 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 msgr2=0x7fea20005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 --2- 172.21.15.138:0/1452978505 >> v2:172.21.15.138:3300/0 conn(0x7fea200036b0 0x7fea20005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: _finish_hunting 0 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 monclient: found mon.noname-c-legacy 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.190:6789/0 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2000af90 con 0x7fea20006160 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: _finish_auth 0 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7fea2400f040 con 0x7fea20006160 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-11T05:20:37.271 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: dump: 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] but i'm connected to v1:172.21.15.190:6789/0, reconnecting 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:37.272 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 >> v1:172.21.15.190:6789/0 conn(0x7fea20006160 legacy=0x7fea20008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea1c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 reap_dead start 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient(hunting): picked mon.c con 0x7fea3413da40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c0040b0 0x7fea1c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient(hunting): picked mon.b con 0x7fea1c0040b0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea1c004e20 0x7fea1c0052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient(hunting): picked mon.a con 0x7fea1c004e20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fea340a30f0 con 0x7fea1c004e20 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c00bd30 con 0x7fea1c0040b0 2024-09-11T05:20:37.273 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fea1c00bf30 con 0x7fea3413da40 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea31ffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea337fe640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea1c004e20 msgr2=0x7fea1c0052a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea337fe640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea1c004e20 0x7fea1c0052a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea1c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient(hunting): get_auth_request con 0x7fea3413da40 auth_method 0 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea327fc640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c0040b0 0x7fea1c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea327fc640 10 monclient(hunting): get_auth_request con 0x7fea1c0040b0 auth_method 0 2024-09-11T05:20:37.274 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea327fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient(hunting): handle_auth_done global_id 104137 payload 293 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c0040b0 msgr2=0x7fea1c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c0040b0 0x7fea1c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea1c004e20 msgr2=0x7fea1c0052a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:37.275 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea1c004e20 0x7fea1c0052a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 1 monclient: found mon.c 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c003530 con 0x7fea3413da40 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient: _finish_auth 0 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea393a9640 5 monclient: authenticate success, global_id 104137 2024-09-11T05:20:37.276 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.270+0000 7fea393a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:37.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea32ffd640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea1c0039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.279 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea1c0039c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea28007d60 tx=0x7fea2801caa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d1cd554ab829646 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28007a10 con 0x7fea3413da40 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea31ffb640 10 monclient: dump: 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.280 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea393a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea280a5b10 con 0x7fea3413da40 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea31ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea31ffb640 1 -- 172.21.15.138:0/1452978505 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea280ad380 con 0x7fea3413da40 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.278+0000 7fea31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea31ffb640 10 monclient: dump: 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.281 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.282 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.282 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.282 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.282 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.282 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:37.282 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 4 set_mon_vals no callback set 2024-09-11T05:20:37.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 10 monclient: shutdown 2024-09-11T05:20:37.283 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 -- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 msgr2=0x7fea1c0039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea1c0039c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea28007d60 tx=0x7fea2801caa0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 -- 172.21.15.138:0/1452978505 shutdown_connections 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea1c004e20 0x7fea1c0052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c0040b0 0x7fea1c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 --2- 172.21.15.138:0/1452978505 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea3413da40 0x7fea1c0039c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 -- 172.21.15.138:0/1452978505 >> 172.21.15.138:0/1452978505 conn(0x7fea340ab940 msgr2=0x7fea340a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 -- 172.21.15.138:0/1452978505 shutdown_connections 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 -- 172.21.15.138:0/1452978505 wait complete. 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 10 monclient: build_initial_monmap 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 10 monclient: monmap: 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:37.284 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 librados: starting msgr at 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 librados: starting objecter 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.282+0000 7fea393a9640 1 Processor -- start 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 -- start start 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 librados: setting wanted keys 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 librados: calling monclient init 2024-09-11T05:20:37.285 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient: init 2024-09-11T05:20:37.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:37.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:20:37.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea1c0040b0 0x7fea341ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient(hunting): picked mon.noname-c con 0x7fea1c0040b0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:37.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c004e20 0x7fea3413ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient(hunting): picked mon.noname-b con 0x7fea1c004e20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea3413da40 0x7fea3413cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient(hunting): picked mon.noname-a con 0x7fea3413da40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fea340a3c30 con 0x7fea3413da40 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea340a3ab0 con 0x7fea1c004e20 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fea340a3db0 con 0x7fea1c0040b0 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea393a9640 10 monclient(hunting): authenticate will time out at 1720.725227s 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea337fe640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea3413da40 msgr2=0x7fea3413cfd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea337fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea3413da40 0x7fea3413cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea1c0040b0 0x7fea341ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea1c0040b0 0x7fea341ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:33214/0 (socket says 172.21.15.138:33214) 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 -- 172.21.15.138:0/1437466243 learned_addr learned my addr 172.21.15.138:0/1437466243 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient(hunting): get_auth_request con 0x7fea1c0040b0 auth_method 0 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea327fc640 1 --2- 172.21.15.138:0/1437466243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c004e20 0x7fea3413ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea327fc640 10 monclient(hunting): get_auth_request con 0x7fea1c004e20 auth_method 0 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient(hunting): handle_auth_done global_id 84368 payload 824 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 -- 172.21.15.138:0/1437466243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c004e20 msgr2=0x7fea3413ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1437466243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c004e20 0x7fea3413ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 -- 172.21.15.138:0/1437466243 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea3413da40 msgr2=0x7fea3413cfd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1437466243 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea3413da40 0x7fea3413cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 monclient: found mon.noname-c 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.286+0000 7fea32ffd640 1 -- 172.21.15.138:0/1437466243 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea340a6ac0 con 0x7fea1c0040b0 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea32ffd640 1 -- 172.21.15.138:0/1437466243 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea1c0040b0 msgr2=0x7fea341ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea32ffd640 1 --2- 172.21.15.138:0/1437466243 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea1c0040b0 0x7fea341ccc40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea28098630 tx=0x7fea1c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 client.?.objecter ms_handle_connect 0x7fea1c0040b0 2024-09-11T05:20:37.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 1 -- 172.21.15.138:0/1437466243 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea280ae420 con 0x7fea1c0040b0 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: dump: 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.294 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 1 -- 172.21.15.138:0/1437466243 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea280ae5c0 con 0x7fea1c0040b0 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 5 monclient: authenticate success, global_id 84368 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 10 monclient: _renew_subs 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea340a23d0 con 0x7fea1c0040b0 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 1 -- 172.21.15.138:0/1437466243 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea280ae990 con 0x7fea1c0040b0 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 monclient: dump: 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.295 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea317fa640 4 set_mon_vals no callback set 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 10 client.84368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 10 monclient: _renew_subs 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea34129440 con 0x7fea1c0040b0 2024-09-11T05:20:37.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 1 librados: init done 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea340a4610 con 0x7fea1c0040b0 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 1 -- 172.21.15.138:0/1437466243 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fea280aeb30 con 0x7fea1c0040b0 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 1 --2- 172.21.15.138:0/1437466243 >> v2:172.21.15.190:6800/1562782189 conn(0x7fea2003d7a0 0x7fea2003fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 1 -- 172.21.15.138:0/1437466243 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(523..523 src has 269..523) ==== 5845+0+0 (secure 0 0 0) 0x7fea280e9540 con 0x7fea1c0040b0 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 10 client.84368.objecter ms_dispatch 0x7fea34000ba0 osd_map(523..523 src has 269..523) 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 3 client.84368.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 3 client.84368.objecter handle_osd_map decoding full epoch 523 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea1b7fe640 20 client.84368.objecter dump_active .. 0 homeless 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea327fc640 1 --2- 172.21.15.138:0/1437466243 >> v2:172.21.15.190:6800/1562782189 conn(0x7fea2003d7a0 0x7fea2003fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.290+0000 7fea327fc640 10 monclient: get_auth_request con 0x7fea2003d7a0 auth_method 0 2024-09-11T05:20:37.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.294+0000 7fea1b7fe640 10 client.84368.objecter ms_handle_connect 0x7fea2003d7a0 2024-09-11T05:20:37.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.294+0000 7fea327fc640 1 --2- 172.21.15.138:0/1437466243 >> v2:172.21.15.190:6800/1562782189 conn(0x7fea2003d7a0 0x7fea2003fc60 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7fea34128ad0 tx=0x7fea2400e950 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.298+0000 7fea1b7fe640 1 -- 172.21.15.138:0/1437466243 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fea28016800 con 0x7fea1c0040b0 2024-09-11T05:20:37.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.298+0000 7fea1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:37.298 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.298+0000 7fea1b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:37.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea393a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:20:37.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea393a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:37.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea393a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea34128ba0 con 0x7fea1c0040b0 2024-09-11T05:20:37.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea1b7fe640 1 -- 172.21.15.138:0/1437466243 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fea280a59d0 con 0x7fea1c0040b0 2024-09-11T05:20:37.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:20:37.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea1b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:37.522 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:20:37.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea393a9640 10 librados: watch_flush enter 2024-09-11T05:20:37.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea393a9640 10 librados: watch_flush exit 2024-09-11T05:20:37.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea393a9640 20 client.84368.objecter shutdown clearing up homeless session... 2024-09-11T05:20:37.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.522+0000 7fea393a9640 10 client.84368.objecter successfully canceled tick 2024-09-11T05:20:37.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 >> v2:172.21.15.190:6800/1562782189 conn(0x7fea2003d7a0 msgr2=0x7fea2003fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 --2- 172.21.15.138:0/1437466243 >> v2:172.21.15.190:6800/1562782189 conn(0x7fea2003d7a0 0x7fea2003fc60 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7fea34128ad0 tx=0x7fea2400e950 comp rx=0 tx=0).stop 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 10 monclient: shutdown 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea1c0040b0 msgr2=0x7fea341ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 --2- 172.21.15.138:0/1437466243 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea1c0040b0 0x7fea341ccc40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea28098630 tx=0x7fea1c005840 comp rx=0 tx=0).stop 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 shutdown_connections 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 --2- 172.21.15.138:0/1437466243 >> v2:172.21.15.190:6800/1562782189 conn(0x7fea2003d7a0 0x7fea2003fc60 unknown :-1 s=CLOSED pgs=3466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 --2- 172.21.15.138:0/1437466243 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fea3413da40 0x7fea3413cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 --2- 172.21.15.138:0/1437466243 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea1c004e20 0x7fea3413ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 --2- 172.21.15.138:0/1437466243 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fea1c0040b0 0x7fea341ccc40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 >> 172.21.15.138:0/1437466243 conn(0x7fea340ab940 msgr2=0x7fea340a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 shutdown_connections 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 -- 172.21.15.138:0/1437466243 wait complete. 2024-09-11T05:20:37.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.526+0000 7fea393a9640 1 librados: shutdown 2024-09-11T05:20:37.539 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:20:37.539 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T05:20:37.540 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient: get_monmap_and_config 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient: build_initial_monmap 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient: monmap: 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:37.665645+0000 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:37.665645+0000 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:37.660 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.661 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:37.661 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:37.661 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:37.661 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:37.661 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:37.661 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 1 Processor -- start 2024-09-11T05:20:37.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 1 -- start start 2024-09-11T05:20:37.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient: init 2024-09-11T05:20:37.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:37.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-11T05:20:37.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c4c09da20 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:37.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c4c0a2030 addr v1:172.21.15.138:6789/0 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a24a0 0x7f1c4c13ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4c0a24a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4c064590 con 0x7f1c4c0a2030 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4c0a2c70 con 0x7f1c4c09da20 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1c4c09de00 con 0x7f1c4c0a24a0 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c51c1f640 1 -- >> v1:172.21.15.138:6789/0 conn(0x7f1c4c0a2030 legacy=0x7f1c4c13b520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.138:6789/0 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c54684640 10 monclient(hunting): authenticate will time out at 1721.101931s 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a24a0 0x7f1c4c13ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a24a0 0x7f1c4c13ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:47484/0 (socket says 172.21.15.138:47484) 2024-09-11T05:20:37.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 1 -- 172.21.15.138:0/1976481628 learned_addr learned my addr 172.21.15.138:0/1976481628 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient(hunting): get_auth_request con 0x7f1c4c0a24a0 auth_method 0 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient(hunting): handle_auth_done global_id 84371 payload 274 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 1 -- 172.21.15.138:0/1976481628 >> v1:172.21.15.190:6789/0 conn(0x7f1c4c09da20 legacy=0x7f1c4c0a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 1 -- 172.21.15.138:0/1976481628 >> v1:172.21.15.138:6789/0 conn(0x7f1c4c0a2030 legacy=0x7f1c4c13b520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient: _finish_hunting 0 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 1 monclient: found mon.noname-b 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:37.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.662+0000 7f1c52c21640 1 -- 172.21.15.138:0/1976481628 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4c0a2f50 con 0x7f1c4c0a24a0 2024-09-11T05:20:37.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c52c21640 1 -- 172.21.15.138:0/1976481628 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a24a0 msgr2=0x7f1c4c13ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c52c21640 1 --2- 172.21.15.138:0/1976481628 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a24a0 0x7f1c4c13ec50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c06d670 tx=0x7f1c3c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f28a4f7a62aa73d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 1 -- 172.21.15.138:0/1976481628 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c0b2070 con 0x7f1c4c0a24a0 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: dump: 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.668 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: _finish_auth 0 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: _check_auth_tickets 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 5 monclient: authenticate success, global_id 84371 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 1 -- 172.21.15.138:0/1976481628 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c3c0ad680 con 0x7f1c4c0a24a0 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 1 -- 172.21.15.138:0/1976481628 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c0ada50 con 0x7f1c4c0a24a0 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 4 set_mon_vals no callback set 2024-09-11T05:20:37.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c5141e640 10 monclient: dump: 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 10 monclient: shutdown 2024-09-11T05:20:37.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 1 -- 172.21.15.138:0/1976481628 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a24a0 msgr2=0x7f1c4c13ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 1 --2- 172.21.15.138:0/1976481628 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a24a0 0x7f1c4c13ec50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c06d670 tx=0x7f1c3c01c440 comp rx=0 tx=0).stop 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 1 -- 172.21.15.138:0/1976481628 shutdown_connections 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 1 --2- 172.21.15.138:0/1976481628 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a24a0 0x7f1c4c13ec50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 1 -- 172.21.15.138:0/1976481628 >> 172.21.15.138:0/1976481628 conn(0x7f1c4c063640 msgr2=0x7f1c4c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 1 -- 172.21.15.138:0/1976481628 shutdown_connections 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.666+0000 7f1c54684640 1 -- 172.21.15.138:0/1976481628 wait complete. 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: build_initial_monmap 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: monmap: 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:37.671 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 librados: starting msgr at 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 librados: starting objecter 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 Processor -- start 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 -- start start 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 librados: setting wanted keys 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 librados: calling monclient init 2024-09-11T05:20:37.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: init 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c09da20 0x7f1c4c1d0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient(hunting): picked mon.noname-a con 0x7f1c4c09da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a2030 0x7f1c4c13e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4c0a2030 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c4c0a24a0 0x7f1c4c13ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4c0a24a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c141dc0 con 0x7f1c4c09da20 2024-09-11T05:20:37.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4c141c40 con 0x7f1c4c0a24a0 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1c4c141f40 con 0x7f1c4c0a2030 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient(hunting): authenticate will time out at 1721.108996s 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c52420640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c09da20 msgr2=0x7f1c4c1d0480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c52420640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c09da20 0x7f1c4c1d0480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a2030 0x7f1c4c13e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a2030 0x7f1c4c13e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:47490/0 (socket says 172.21.15.138:47490) 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 -- 172.21.15.138:0/2251135282 learned_addr learned my addr 172.21.15.138:0/2251135282 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient(hunting): get_auth_request con 0x7f1c4c0a2030 auth_method 0 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient(hunting): handle_auth_done global_id 84374 payload 824 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c4c0a24a0 msgr2=0x7f1c4c13ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c4c0a24a0 0x7f1c4c13ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c09da20 msgr2=0x7f1c4c1d0480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1c4c09da20 0x7f1c4c1d0480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient: _finish_hunting 0 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 monclient: found mon.noname-c 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 -- 172.21.15.138:0/2251135282 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4c13ac50 con 0x7f1c4c0a2030 2024-09-11T05:20:37.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a2030 msgr2=0x7f1c4c13e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 client.?.objecter ms_handle_connect 0x7f1c4c0a2030 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c51c1f640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a2030 0x7f1c4c13e4e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000ba10 tx=0x7f1c4000bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4000cd00 con 0x7f1c4c0a2030 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: dump: 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.680 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: _finish_auth 0 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: _check_auth_tickets 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c40010070 con 0x7f1c4c0a2030 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 5 monclient: authenticate success, global_id 84374 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: _renew_subs 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4c073c80 con 0x7f1c4c0a2030 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c400156b0 con 0x7f1c4c0a2030 2024-09-11T05:20:37.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c4affd640 10 monclient: dump: 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c50c1d640 4 set_mon_vals no callback set 2024-09-11T05:20:37.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 client.84374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: _renew_subs 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4c13cfb0 con 0x7f1c4c0a2030 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.670+0000 7f1c54684640 1 librados: init done 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c54684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c54684640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139190 0x7f1c4c139570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c54684640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1c4c0a19a0 con 0x7f1c4c139190 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139190 0x7f1c4c139570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 10 monclient: get_auth_request con 0x7f1c4c139190 auth_method 0 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1c4001e3f0 con 0x7f1c4c0a2030 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 10 monclient(hunting): handle_auth_done global_id 84377 payload 824 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 1 -- 172.21.15.138:0/2251135282 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c4c0008e0 con 0x7f1c4c139190 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 --2- 172.21.15.138:0/2251135282 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1c3803ce50 0x7f1c3803f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(524..524 src has 269..524) ==== 5845+0+0 (secure 0 0 0) 0x7f1c4004fea0 con 0x7f1c4c0a2030 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 10 client.84374.objecter ms_dispatch 0x7f1c4c000ba0 osd_map(524..524 src has 269..524) 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 3 client.84374.objecter handle_osd_map got epochs [524,524] > 0 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 3 client.84374.objecter handle_osd_map decoding full epoch 524 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 20 client.84374.objecter dump_active .. 0 homeless 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52c21640 1 --2- 172.21.15.138:0/2251135282 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1c3803ce50 0x7f1c3803f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52c21640 10 monclient: get_auth_request con 0x7f1c3803ce50 auth_method 0 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 10 client.84374.objecter ms_handle_connect 0x7f1c4c139190 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 10 client.84374.objecter resend_mon_ops 2024-09-11T05:20:37.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139190 msgr2=0x7f1c4c139570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c52420640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139190 0x7f1c4c139570 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1c44007fd0 tx=0x7f1c4400c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44017070 con 0x7f1c4c139190 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 10 monclient: dump: 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.685 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1c440057d0 con 0x7f1c4c139190 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139190 msgr2=0x7f1c4c139570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139190 0x7f1c4c139570 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1c44007fd0 tx=0x7f1c4400c810 comp rx=0 tx=0).stop 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139190 msgr2=0x7f1c4c139570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.674+0000 7f1c4affd640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139190 0x7f1c4c139570 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1c44007fd0 tx=0x7f1c4400c810 comp rx=0 tx=0).stop 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.678+0000 7f1c52c21640 1 --2- 172.21.15.138:0/2251135282 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1c3803ce50 0x7f1c3803f310 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c09e050 tx=0x7f1c3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.678+0000 7f1c4affd640 10 client.84374.objecter ms_handle_connect 0x7f1c3803ce50 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c54684640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c54684640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139ab0 0x7f1c4c133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c54684640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1c4c13b7b0 con 0x7f1c4c139ab0 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139ab0 0x7f1c4c133df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 10 monclient: get_auth_request con 0x7f1c4c139ab0 auth_method 0 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 10 monclient(hunting): handle_auth_done global_id 84380 payload 824 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.682+0000 7f1c52420640 1 -- 172.21.15.138:0/2251135282 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c4c13c710 con 0x7f1c4c139ab0 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c52420640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139ab0 msgr2=0x7f1c4c133df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 10 client.84374.objecter ms_handle_connect 0x7f1c4c139ab0 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 10 client.84374.objecter resend_mon_ops 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c52420640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139ab0 0x7f1c4c133df0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1c44014000 tx=0x7f1c440037f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44017040 con 0x7f1c4c139ab0 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 10 monclient: dump: 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f1c44019ab0 con 0x7f1c4c139ab0 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139ab0 msgr2=0x7f1c4c133df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139ab0 0x7f1c4c133df0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1c44014000 tx=0x7f1c440037f0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139ab0 msgr2=0x7f1c4c133df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c4affd640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c139ab0 0x7f1c4c133df0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1c44014000 tx=0x7f1c440037f0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:20:37.691 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 54, 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 281456, 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:37.692 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:37.693 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:20:37.694 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:37.695 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.696 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:37.697 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:37.698 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:20:37.699 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.700 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:20:37.701 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.702 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:20:37.703 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:37.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 10 librados: watch_flush enter 2024-09-11T05:20:37.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 10 librados: watch_flush exit 2024-09-11T05:20:37.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 20 client.84374.objecter shutdown clearing up homeless session... 2024-09-11T05:20:37.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 10 client.84374.objecter successfully canceled tick 2024-09-11T05:20:37.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1c3803ce50 msgr2=0x7f1c3803f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 --2- 172.21.15.138:0/2251135282 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1c3803ce50 0x7f1c3803f310 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c09e050 tx=0x7f1c3c008040 comp rx=0 tx=0).stop 2024-09-11T05:20:37.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 10 monclient: shutdown 2024-09-11T05:20:37.705 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a2030 msgr2=0x7f1c4c13e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 --2- 172.21.15.138:0/2251135282 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1c4c0a2030 0x7f1c4c13e4e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000ba10 tx=0x7f1c4000bed0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c52c21640 1 -- 172.21.15.138:0/2251135282 reap_dead start 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c52c21640 1 -- 172.21.15.138:0/2251135282 reap_dead start 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 shutdown_connections 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 >> 172.21.15.138:0/2251135282 conn(0x7f1c4c063640 msgr2=0x7f1c4c132f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 shutdown_connections 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 -- 172.21.15.138:0/2251135282 wait complete. 2024-09-11T05:20:37.706 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.686+0000 7f1c54684640 1 librados: shutdown 2024-09-11T05:20:37.706 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:20:37.706 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-11T05:20:37.707 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:20:37.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient: get_monmap_and_config 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient: build_initial_monmap 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient: monmap: 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:20:37.822311+0000 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:20:37.822311+0000 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:20:37.817 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:20:37.818 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:20:37.818 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 1 Processor -- start 2024-09-11T05:20:37.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 1 -- start start 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient: init 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff510138690 addr v1:172.21.15.190:6789/0 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff5100a34f0 0x7ff51013d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5100a34f0 addr v2:172.21.15.138:3300/0 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51013dbb0 0x7ff51013ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient(hunting): picked mon.noname-b con 0x7ff51013dbb0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:37.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5100a24a0 con 0x7ff510138690 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7ff5100a3160 con 0x7ff5100a34f0 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff5100a2780 con 0x7ff51013dbb0 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff51526d640 10 monclient(hunting): authenticate will time out at 1721.258207s 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50e7fc640 1 -- >> v2:172.21.15.138:3300/0 conn(0x7ff5100a34f0 msgr2=0x7ff51013d670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50e7fc640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff5100a34f0 0x7ff51013d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51013dbb0 0x7ff51013ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51013dbb0 0x7ff51013ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:47516/0 (socket says 172.21.15.138:47516) 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1478887729 learned_addr learned my addr 172.21.15.138:0/1478887729 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient(hunting): get_auth_request con 0x7ff51013dbb0 auth_method 0 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_done global_id 84383 payload 274 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1478887729 >> v2:172.21.15.138:3300/0 conn(0x7ff5100a34f0 msgr2=0x7ff51013d670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/1478887729 >> v2:172.21.15.138:3300/0 conn(0x7ff5100a34f0 0x7ff51013d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1478887729 >> v1:172.21.15.190:6789/0 conn(0x7ff510138690 legacy=0x7ff510138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 1 monclient: found mon.noname-b 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.818+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1478887729 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5100a2a60 con 0x7ff51013dbb0 2024-09-11T05:20:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1478887729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51013dbb0 msgr2=0x7ff51013ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/1478887729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51013dbb0 0x7ff51013ffa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff508009e50 tx=0x7ff50801c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4cf9f8a82ab893 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 1 -- 172.21.15.138:0/1478887729 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5080a5770 con 0x7ff51013dbb0 2024-09-11T05:20:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: dump: 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.824 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: _finish_auth 0 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: _check_auth_tickets 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 1 -- 172.21.15.138:0/1478887729 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5080a5910 con 0x7ff51013dbb0 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff51526d640 5 monclient: authenticate success, global_id 84383 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff51526d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff51526d640 10 monclient: get_monmap_and_config success 2024-09-11T05:20:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 1 -- 172.21.15.138:0/1478887729 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5080a5ce0 con 0x7ff51013dbb0 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff51526d640 4 set_mon_vals no callback set 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.822+0000 7ff50dffb640 10 monclient: dump: 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.826 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient: shutdown 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- 172.21.15.138:0/1478887729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51013dbb0 msgr2=0x7ff51013ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 --2- 172.21.15.138:0/1478887729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51013dbb0 0x7ff51013ffa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff508009e50 tx=0x7ff50801c910 comp rx=0 tx=0).stop 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- 172.21.15.138:0/1478887729 shutdown_connections 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 --2- 172.21.15.138:0/1478887729 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51013dbb0 0x7ff51013ffa0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 --2- 172.21.15.138:0/1478887729 >> v2:172.21.15.138:3300/0 conn(0x7ff5100a34f0 0x7ff51013d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- 172.21.15.138:0/1478887729 >> 172.21.15.138:0/1478887729 conn(0x7ff510063640 msgr2=0x7ff510063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- 172.21.15.138:0/1478887729 shutdown_connections 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- 172.21.15.138:0/1478887729 wait complete. 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient: build_initial_monmap 2024-09-11T05:20:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient: monmap: 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 librados: starting msgr at 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 librados: starting objecter 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 Processor -- start 2024-09-11T05:20:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- start start 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 librados: setting wanted keys 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 librados: calling monclient init 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient: init 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient: _reopen_session rank -1 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5100a34f0 0x7ff51012e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5100a34f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff510138690 0x7ff51012ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient(hunting): picked mon.noname-c con 0x7ff510138690 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51013dbb0 0x7ff5101d0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient(hunting): picked mon.noname-b con 0x7ff51013dbb0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff510142d00 con 0x7ff5100a34f0 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff510142e80 con 0x7ff51013dbb0 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff510143000 con 0x7ff510138690 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient(hunting): _renew_subs 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff51526d640 10 monclient(hunting): authenticate will time out at 1721.266277s 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50effd640 1 -- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5100a34f0 msgr2=0x7ff51012e9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.138:3300/0 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5100a34f0 0x7ff51012e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff510138690 0x7ff51012ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff510138690 0x7ff51012ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:47528/0 (socket says 172.21.15.138:47528) 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 -- 172.21.15.138:0/822167285 learned_addr learned my addr 172.21.15.138:0/822167285 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient(hunting): get_auth_request con 0x7ff510138690 auth_method 0 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient(hunting): handle_auth_done global_id 84386 payload 824 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51013dbb0 msgr2=0x7ff5101d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51013dbb0 0x7ff5101d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5100a34f0 msgr2=0x7ff51012e9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff5100a34f0 0x7ff51012e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient: _finish_hunting 0 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 monclient: found mon.noname-c 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:20:37.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.826+0000 7ff50e7fc640 1 -- 172.21.15.138:0/822167285 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5101323a0 con 0x7ff510138690 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50e7fc640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff510138690 msgr2=0x7ff51012ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50e7fc640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff510138690 0x7ff51012ef10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff50400b4d0 tx=0x7ff50400b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 client.?.objecter ms_handle_connect 0x7ff510138690 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504005070 con 0x7ff510138690 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:20:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: dump: 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: _finish_auth 0 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: _check_auth_tickets 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:20:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff504015070 con 0x7ff510138690 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 5 monclient: authenticate success, global_id 84386 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 10 monclient: _renew_subs 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff504005450 con 0x7ff510138690 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5100a61a0 con 0x7ff510138690 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50d7fa640 4 set_mon_vals no callback set 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 monclient: dump: 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 10 client.84386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 10 monclient: _renew_subs 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5101357a0 con 0x7ff510138690 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 1 librados: init done 2024-09-11T05:20:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002d450 0x7ff51002d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff510138500 con 0x7ff51002d450 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7ff5040055f0 con 0x7ff510138690 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002d450 0x7ff51002d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 10 monclient: get_auth_request con 0x7ff51002d450 auth_method 0 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 1 --2- 172.21.15.138:0/822167285 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4fc03cea0 0x7ff4fc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(524..524 src has 269..524) ==== 5845+0+0 (secure 0 0 0) 0x7ff50404fa80 con 0x7ff510138690 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 10 client.84386.objecter ms_dispatch 0x7ff510000ba0 osd_map(524..524 src has 269..524) 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 3 client.84386.objecter handle_osd_map got epochs [524,524] > 0 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 3 client.84386.objecter handle_osd_map decoding full epoch 524 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff4f77fe640 20 client.84386.objecter dump_active .. 0 homeless 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/822167285 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4fc03cea0 0x7ff4fc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50f7fe640 10 monclient: get_auth_request con 0x7ff4fc03cea0 auth_method 0 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 10 monclient(hunting): handle_auth_done global_id 104143 payload 824 2024-09-11T05:20:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.830+0000 7ff50effd640 1 -- 172.21.15.138:0/822167285 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff510138af0 con 0x7ff51002d450 2024-09-11T05:20:37.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 10 client.84386.objecter ms_handle_connect 0x7ff51002d450 2024-09-11T05:20:37.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 10 client.84386.objecter resend_mon_ops 2024-09-11T05:20:37.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff50effd640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002d450 msgr2=0x7ff51002d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff50effd640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002d450 0x7ff51002d830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff500007fd0 tx=0x7ff50000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/822167285 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4fc03cea0 0x7ff4fc03f360 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7ff5100a7f50 tx=0x7ff508004ac0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 10 client.84386.objecter ms_handle_connect 0x7ff4fc03cea0 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500017070 con 0x7ff51002d450 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 10 monclient: dump: 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.845 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff5000057d0 con 0x7ff51002d450 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002d450 msgr2=0x7ff51002d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002d450 0x7ff51002d830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff500007fd0 tx=0x7ff50000c810 comp rx=0 tx=0).stop 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002d450 msgr2=0x7ff51002d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.834+0000 7ff4f77fe640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002d450 0x7ff51002d830 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff500007fd0 tx=0x7ff50000c810 comp rx=0 tx=0).stop 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff51526d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff51526d640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002dd70 0x7ff51002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff51526d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff51009fa30 con 0x7ff51002dd70 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002dd70 0x7ff51002e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:20:37.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 10 monclient: get_auth_request con 0x7ff51002dd70 auth_method 0 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 10 monclient(hunting): handle_auth_done global_id 104146 payload 824 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.842+0000 7ff50effd640 1 -- 172.21.15.138:0/822167285 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff51013d030 con 0x7ff51002dd70 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff50effd640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002dd70 msgr2=0x7ff51002e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 10 client.84386.objecter ms_handle_connect 0x7ff51002dd70 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 10 client.84386.objecter resend_mon_ops 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff50effd640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002dd70 0x7ff51002e180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff500014000 tx=0x7ff5000037f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500017040 con 0x7ff51002dd70 2024-09-11T05:20:37.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 10 monclient: dump: 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:20:37.848 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:20:37.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7ff500022400 con 0x7ff51002dd70 2024-09-11T05:20:37.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:20:37.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:20:37.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002dd70 msgr2=0x7ff51002e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002dd70 0x7ff51002e180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff500014000 tx=0x7ff5000037f0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002dd70 msgr2=0x7ff51002e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:20:37.849 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff4f77fe640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff51002dd70 0x7ff51002e180 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff500014000 tx=0x7ff5000037f0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 54, 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 28007, 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.850 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:37.851 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:37.852 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:20:37.853 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:20:37.854 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.855 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:20:37.856 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:20:37.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:20:37.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:20:37.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:20:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:20:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:20:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:20:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:20:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:20:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:20:37.862 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:20:37.862 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:20:37.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 10 librados: watch_flush enter 2024-09-11T05:20:37.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 10 librados: watch_flush exit 2024-09-11T05:20:37.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 20 client.84386.objecter shutdown clearing up homeless session... 2024-09-11T05:20:37.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 10 client.84386.objecter successfully canceled tick 2024-09-11T05:20:37.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4fc03cea0 msgr2=0x7ff4fc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 --2- 172.21.15.138:0/822167285 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4fc03cea0 0x7ff4fc03f360 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7ff5100a7f50 tx=0x7ff508004ac0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 10 monclient: shutdown 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff510138690 msgr2=0x7ff51012ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff50f7fe640 1 -- 172.21.15.138:0/822167285 reap_dead start 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff510138690 0x7ff51012ef10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff50400b4d0 tx=0x7ff50400b990 comp rx=0 tx=0).stop 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 shutdown_connections 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 --2- 172.21.15.138:0/822167285 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff510138690 0x7ff51012ef10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 >> 172.21.15.138:0/822167285 conn(0x7ff510063640 msgr2=0x7ff5100a5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 shutdown_connections 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 -- 172.21.15.138:0/822167285 wait complete. 2024-09-11T05:20:37.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:20:37.846+0000 7ff51526d640 1 librados: shutdown 2024-09-11T05:20:37.865 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:20:37.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:21:47.943 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8983 ms) 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6063 ms) 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (4956 ms) 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (43194 ms) 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13559 ms) 2024-09-11T05:21:47.944 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12020 ms) 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 00000012 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12015 ms) 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6455 ms) 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (623603 ms total) 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-11T05:21:47.945 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1037 ms) 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10985 ms) 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17963 ms) 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (4 ms) 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (29989 ms total) 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (12796 ms) 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-11T05:21:47.946 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (22488 ms) 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: waiting for scrub... 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: done waiting 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24553 ms) 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10026 ms) 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7980 ms) 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7983 ms) 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10427 ms) 2024-09-11T05:21:47.947 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-11T05:22:07.868 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T05:22:07.868 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T05:22:07.869 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T05:22:07.872 INFO:tasks.ceph.mon.a:Started 2024-09-11T05:22:07.872 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:22:07.872 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:07.973 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:07.976+0000 7fe1a16dbe00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 545 -> 0 (sync_force? mon rebuild?) 2024-09-11T05:22:08.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient: get_monmap_and_config 2024-09-11T05:22:08.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient: build_initial_monmap 2024-09-11T05:22:08.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient: monmap: 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:08.010254+0000 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:08.010254+0000 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:08.005 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:08.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 1 Processor -- start 2024-09-11T05:22:08.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 1 -- start start 2024-09-11T05:22:08.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient: init 2024-09-11T05:22:08.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f703c06e0e0 0x7f703c09e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient(hunting): picked mon.noname-c con 0x7f703c06e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c09e960 0x7f703c09ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient(hunting): picked mon.noname-b con 0x7f703c09e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f703c1356f0 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f703c06e4c0 con 0x7f703c1356f0 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f703c09f820 con 0x7f703c06e0e0 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f703c09f540 con 0x7f703c09e960 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f7042a14640 10 monclient(hunting): authenticate will time out at 1811.446025s 2024-09-11T05:22:08.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c09e960 0x7f703c09ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c09e960 0x7f703c09ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55104/0 (socket says 172.21.15.138:55104) 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 1 -- 172.21.15.138:0/2221522291 learned_addr learned my addr 172.21.15.138:0/2221522291 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient(hunting): get_auth_request con 0x7f703c09e960 auth_method 0 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703bfff640 1 --2- 172.21.15.138:0/2221522291 >> v2:172.21.15.190:3300/0 conn(0x7f703c06e0e0 0x7f703c09e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient(hunting): handle_auth_done global_id 84389 payload 274 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 1 -- 172.21.15.138:0/2221522291 >> v2:172.21.15.190:3300/0 conn(0x7f703c06e0e0 msgr2=0x7f703c09e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 1 --2- 172.21.15.138:0/2221522291 >> v2:172.21.15.190:3300/0 conn(0x7f703c06e0e0 0x7f703c09e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 1 -- 172.21.15.138:0/2221522291 >> v1:172.21.15.190:6789/0 conn(0x7f703c1356f0 legacy=0x7f703c135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:08.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:22:08.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 1 monclient: found mon.noname-b 2024-09-11T05:22:08.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:08.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:08.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703b7fe640 1 -- 172.21.15.138:0/2221522291 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703c09fb00 con 0x7f703c09e960 2024-09-11T05:22:08.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703bfff640 10 monclient: get_auth_request con 0x7f703c06e0e0 auth_method 0 2024-09-11T05:22:08.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.008+0000 7f703bfff640 1 --2- 172.21.15.138:0/2221522291 >> v2:172.21.15.190:3300/0 conn(0x7f703c06e0e0 0x7f703c09e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:22:08.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703b7fe640 1 -- 172.21.15.138:0/2221522291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c09e960 msgr2=0x7f703c09ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:08.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703b7fe640 1 --2- 172.21.15.138:0/2221522291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c09e960 0x7f703c09ed40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f703000bb10 tx=0x7f703001eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b03cf85624bfc83 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:08.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 1 -- 172.21.15.138:0/2221522291 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70300b4070 con 0x7f703c09e960 2024-09-11T05:22:08.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:08.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: dump: 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: _finish_auth 0 2024-09-11T05:22:08.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 5 monclient: authenticate success, global_id 84389 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 1 -- 172.21.15.138:0/2221522291 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70300a7a40 con 0x7f703c09e960 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 1 -- 172.21.15.138:0/2221522291 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70300a7e10 con 0x7f703c09e960 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 4 set_mon_vals no callback set 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f703affd640 10 monclient: dump: 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:08.016 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 10 monclient: shutdown 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 1 -- 172.21.15.138:0/2221522291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c09e960 msgr2=0x7f703c09ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 1 --2- 172.21.15.138:0/2221522291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c09e960 0x7f703c09ed40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f703000bb10 tx=0x7f703001eb80 comp rx=0 tx=0).stop 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 1 -- 172.21.15.138:0/2221522291 shutdown_connections 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 1 --2- 172.21.15.138:0/2221522291 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c09e960 0x7f703c09ed40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 1 --2- 172.21.15.138:0/2221522291 >> v2:172.21.15.190:3300/0 conn(0x7f703c06e0e0 0x7f703c09e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 1 -- 172.21.15.138:0/2221522291 >> 172.21.15.138:0/2221522291 conn(0x7f703c063640 msgr2=0x7f703c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 1 -- 172.21.15.138:0/2221522291 shutdown_connections 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.012+0000 7f7042a14640 1 -- 172.21.15.138:0/2221522291 wait complete. 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient: build_initial_monmap 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient: monmap: 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:08.018 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 librados: starting msgr at 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 librados: starting objecter 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 Processor -- start 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 -- start start 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 librados: setting wanted keys 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 librados: calling monclient init 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient: init 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c06e0e0 0x7f703c1d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient(hunting): picked mon.noname-b con 0x7f703c06e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 0x7f703c134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient(hunting): picked mon.noname-a con 0x7f703c09e960 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:08.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c1356f0 0x7f703c1350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient(hunting): picked mon.noname-c con 0x7f703c1356f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f703c0a2b40 con 0x7f703c09e960 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f703c0a29c0 con 0x7f703c06e0e0 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f703c0a2cc0 con 0x7f703c1356f0 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7042a14640 10 monclient(hunting): authenticate will time out at 1811.453232s 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c1356f0 0x7f703c1350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 0x7f703c134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c1356f0 0x7f703c1350c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55116/0 (socket says 172.21.15.138:55116) 2024-09-11T05:22:08.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 -- 172.21.15.138:0/4125039992 learned_addr learned my addr 172.21.15.138:0/4125039992 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 0x7f703c134b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37402/0 (socket says 172.21.15.138:37402) 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient(hunting): get_auth_request con 0x7f703c1356f0 auth_method 0 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 10 monclient(hunting): get_auth_request con 0x7f703c09e960 auth_method 0 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:08.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 1 -- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 msgr2=0x7f703c134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 1 -- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 msgr2=0x7f703c134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 0x7f703c134b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703bfff640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c06e0e0 0x7f703c1d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703b7fe640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 0x7f703c134b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703bfff640 10 monclient(hunting): get_auth_request con 0x7f703c06e0e0 auth_method 0 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f703bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient(hunting): handle_auth_done global_id 84392 payload 824 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 -- 172.21.15.138:0/4125039992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c06e0e0 msgr2=0x7f703c1d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c06e0e0 0x7f703c1d0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 -- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 msgr2=0x7f703c134b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:08.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 0x7f703c134b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient: _finish_hunting 0 2024-09-11T05:22:08.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 monclient: found mon.noname-c 2024-09-11T05:22:08.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:08.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:08.023 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.016+0000 7f7040fb1640 1 -- 172.21.15.138:0/4125039992 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703c122450 con 0x7f703c1356f0 2024-09-11T05:22:08.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7040fb1640 1 -- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c1356f0 msgr2=0x7f703c1350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:08.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7040fb1640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c1356f0 0x7f703c1350c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f703400c780 tx=0x7f703400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:08.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 client.?.objecter ms_handle_connect 0x7f703c1356f0 2024-09-11T05:22:08.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:08.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 1 -- 172.21.15.138:0/4125039992 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034017070 con 0x7f703c1356f0 2024-09-11T05:22:08.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:08.026 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: dump: 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: _finish_auth 0 2024-09-11T05:22:08.027 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 5 monclient: authenticate success, global_id 84392 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 10 monclient: _renew_subs 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 1 -- 172.21.15.138:0/4125039992 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f703400ce30 con 0x7f703c1356f0 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703c123270 con 0x7f703c1356f0 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 1 -- 172.21.15.138:0/4125039992 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034005290 con 0x7f703c1356f0 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 monclient: dump: 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:08.028 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f703a7fc640 4 set_mon_vals no callback set 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 10 client.84392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 10 monclient: _renew_subs 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f703c121d50 con 0x7f703c1356f0 2024-09-11T05:22:08.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 1 librados: init done 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703c09e420 con 0x7f703c1356f0 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 1 -- 172.21.15.138:0/4125039992 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f70340054b0 con 0x7f703c1356f0 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 1 --2- 172.21.15.138:0/4125039992 >> v2:172.21.15.190:6800/1562782189 conn(0x7f702803cea0 0x7f702803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 1 -- 172.21.15.138:0/4125039992 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(597..597 src has 520..597) ==== 5548+0+0 (secure 0 0 0) 0x7f703404f270 con 0x7f703c1356f0 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 10 client.84392.objecter ms_dispatch 0x7f703c000ba0 osd_map(597..597 src has 520..597) 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 3 client.84392.objecter handle_osd_map got epochs [597,597] > 0 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 3 client.84392.objecter handle_osd_map decoding full epoch 597 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f7038ff9640 20 client.84392.objecter dump_active .. 0 homeless 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f703bfff640 1 --2- 172.21.15.138:0/4125039992 >> v2:172.21.15.190:6800/1562782189 conn(0x7f702803cea0 0x7f702803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:08.030 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.020+0000 7f703bfff640 10 monclient: get_auth_request con 0x7f702803cea0 auth_method 0 2024-09-11T05:22:08.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.024+0000 7f703bfff640 1 --2- 172.21.15.138:0/4125039992 >> v2:172.21.15.190:6800/1562782189 conn(0x7f702803cea0 0x7f702803f360 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7f702c006fd0 tx=0x7f702c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:08.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.024+0000 7f7038ff9640 10 client.84392.objecter ms_handle_connect 0x7f702803cea0 2024-09-11T05:22:08.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.024+0000 7f7038ff9640 1 -- 172.21.15.138:0/4125039992 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f7034019cd0 con 0x7f703c1356f0 2024-09-11T05:22:08.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.024+0000 7f7038ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:08.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.024+0000 7f7038ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:08.259 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.260+0000 7f7042a14640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:08.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.260+0000 7f7042a14640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:08.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.260+0000 7f7042a14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:08.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.260+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f703c09e0e0 con 0x7f703c1356f0 2024-09-11T05:22:08.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7038ff9640 1 -- 172.21.15.138:0/4125039992 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f703c09e0e0 con 0x7f703c1356f0 2024-09-11T05:22:08.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7038ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:08.260 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7038ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:08.261 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:08.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 10 librados: watch_flush enter 2024-09-11T05:22:08.263 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 10 librados: watch_flush exit 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 20 client.84392.objecter shutdown clearing up homeless session... 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 10 client.84392.objecter successfully canceled tick 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 >> v2:172.21.15.190:6800/1562782189 conn(0x7f702803cea0 msgr2=0x7f702803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 --2- 172.21.15.138:0/4125039992 >> v2:172.21.15.190:6800/1562782189 conn(0x7f702803cea0 0x7f702803f360 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7f702c006fd0 tx=0x7f702c008040 comp rx=0 tx=0).stop 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 10 monclient: shutdown 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c1356f0 msgr2=0x7f703c1350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c1356f0 0x7f703c1350c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f703400c780 tx=0x7f703400cc40 comp rx=0 tx=0).stop 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 shutdown_connections 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 --2- 172.21.15.138:0/4125039992 >> v2:172.21.15.190:6800/1562782189 conn(0x7f702803cea0 0x7f702803f360 unknown :-1 s=CLOSED pgs=3471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f703c1356f0 0x7f703c1350c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f703c09e960 0x7f703c134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.264 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 --2- 172.21.15.138:0/4125039992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f703c06e0e0 0x7f703c1d0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:08.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 >> 172.21.15.138:0/4125039992 conn(0x7f703c063640 msgr2=0x7f703c0a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:08.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 shutdown_connections 2024-09-11T05:22:08.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 -- 172.21.15.138:0/4125039992 wait complete. 2024-09-11T05:22:08.265 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:08.264+0000 7f7042a14640 1 librados: shutdown 2024-09-11T05:22:11.281 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient: get_monmap_and_config 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient: build_initial_monmap 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient: monmap: 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:11.414497+0000 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:11.414497+0000 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:11.409 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:11.410 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:11.410 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:11.410 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:11.410 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:11.410 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 1 Processor -- start 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 1 -- start start 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient: init 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 0x7eff4809e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient(hunting): picked mon.noname-a con 0x7eff4806e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff4809e960 0x7eff4809ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient(hunting): picked mon.noname-c con 0x7eff4809e960 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:11.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff48135b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient(hunting): picked mon.noname-b con 0x7eff481356f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7eff480a19b0 con 0x7eff4806e0e0 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7eff4806e4c0 con 0x7eff4809e960 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7eff4809f820 con 0x7eff481356f0 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff507e5640 10 monclient(hunting): authenticate will time out at 1814.851351s 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 0x7eff4809e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff48135b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff48135b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55152/0 (socket says 172.21.15.138:55152) 2024-09-11T05:22:11.414 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 -- 172.21.15.138:0/2212064440 learned_addr learned my addr 172.21.15.138:0/2212064440 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient(hunting): get_auth_request con 0x7eff481356f0 auth_method 0 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 0x7eff4809e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37416/0 (socket says 172.21.15.138:37416) 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 10 monclient(hunting): get_auth_request con 0x7eff4806e0e0 auth_method 0 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 1 -- 172.21.15.138:0/2212064440 >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 msgr2=0x7eff4809e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:22:11.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 1 -- 172.21.15.138:0/2212064440 >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 msgr2=0x7eff4809e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 1 --2- 172.21.15.138:0/2212064440 >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 0x7eff4809e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4e581640 1 --2- 172.21.15.138:0/2212064440 >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 0x7eff4809e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4dd80640 1 --2- 172.21.15.138:0/2212064440 >> v2:172.21.15.190:3300/0 conn(0x7eff4809e960 0x7eff4809ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4dd80640 10 monclient(hunting): get_auth_request con 0x7eff4809e960 auth_method 0 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4dd80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4dd80640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4dd80640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient(hunting): handle_auth_done global_id 84395 payload 274 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 -- 172.21.15.138:0/2212064440 >> v2:172.21.15.190:3300/0 conn(0x7eff4809e960 msgr2=0x7eff4809ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 --2- 172.21.15.138:0/2212064440 >> v2:172.21.15.190:3300/0 conn(0x7eff4809e960 0x7eff4809ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 -- 172.21.15.138:0/2212064440 >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 msgr2=0x7eff4809e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:11.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 --2- 172.21.15.138:0/2212064440 >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 0x7eff4809e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient: _finish_hunting 0 2024-09-11T05:22:11.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 monclient: found mon.noname-b 2024-09-11T05:22:11.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:11.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:11.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.412+0000 7eff4ed82640 1 -- 172.21.15.138:0/2212064440 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4809f540 con 0x7eff481356f0 2024-09-11T05:22:11.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 -- 172.21.15.138:0/2212064440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 msgr2=0x7eff48135b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:11.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 --2- 172.21.15.138:0/2212064440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff48135b20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7eff4000d150 tx=0x7eff4001ea80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=284bb308627fc697 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 1 -- 172.21.15.138:0/2212064440 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4000b2d0 con 0x7eff481356f0 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: dump: 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:11.423 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: _finish_auth 0 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: _check_auth_tickets 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 1 -- 172.21.15.138:0/2212064440 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff4000b470 con 0x7eff481356f0 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 1 -- 172.21.15.138:0/2212064440 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff400a7a50 con 0x7eff481356f0 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 5 monclient: authenticate success, global_id 84395 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 4 set_mon_vals no callback set 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:11.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4d57f640 10 monclient: dump: 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:11.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient: shutdown 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- 172.21.15.138:0/2212064440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 msgr2=0x7eff48135b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 --2- 172.21.15.138:0/2212064440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff48135b20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7eff4000d150 tx=0x7eff4001ea80 comp rx=0 tx=0).stop 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- 172.21.15.138:0/2212064440 shutdown_connections 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 --2- 172.21.15.138:0/2212064440 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff48135b20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 --2- 172.21.15.138:0/2212064440 >> v2:172.21.15.190:3300/0 conn(0x7eff4809e960 0x7eff4809ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 --2- 172.21.15.138:0/2212064440 >> v2:172.21.15.138:3300/0 conn(0x7eff4806e0e0 0x7eff4809e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- 172.21.15.138:0/2212064440 >> 172.21.15.138:0/2212064440 conn(0x7eff48063640 msgr2=0x7eff48063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- 172.21.15.138:0/2212064440 shutdown_connections 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- 172.21.15.138:0/2212064440 wait complete. 2024-09-11T05:22:11.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient: build_initial_monmap 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient: monmap: 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 librados: starting msgr at 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 librados: starting objecter 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 Processor -- start 2024-09-11T05:22:11.427 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- start start 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 librados: setting wanted keys 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 librados: calling monclient init 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient: init 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff4806e0e0 0x7eff48134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient(hunting): picked mon.noname-b con 0x7eff4806e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 0x7eff481345b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient(hunting): picked mon.noname-a con 0x7eff4809e960 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff4811e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient(hunting): picked mon.noname-c con 0x7eff481356f0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:11.428 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7eff480a1430 con 0x7eff4809e960 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7eff480a12b0 con 0x7eff4806e0e0 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7eff480a29c0 con 0x7eff481356f0 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff507e5640 10 monclient(hunting): authenticate will time out at 1814.858497s 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 0x7eff481345b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff4811e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 0x7eff481345b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37420/0 (socket says 172.21.15.138:37420) 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient(hunting): get_auth_request con 0x7eff481356f0 auth_method 0 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 1 -- 172.21.15.138:0/1316704889 learned_addr learned my addr 172.21.15.138:0/1316704889 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:11.429 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 10 monclient(hunting): get_auth_request con 0x7eff4809e960 auth_method 0 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 1 -- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 msgr2=0x7eff481345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 1 -- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 msgr2=0x7eff481345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 0x7eff481345b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4dd80640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 0x7eff481345b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4e581640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff4806e0e0 0x7eff48134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4e581640 10 monclient(hunting): get_auth_request con 0x7eff4806e0e0 auth_method 0 2024-09-11T05:22:11.430 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4e581640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4e581640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4e581640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient(hunting): handle_auth_done global_id 84398 payload 824 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 -- 172.21.15.138:0/1316704889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff4806e0e0 msgr2=0x7eff48134070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff4806e0e0 0x7eff48134070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 -- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 msgr2=0x7eff481345b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 0x7eff481345b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient: _finish_hunting 0 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 monclient: found mon.noname-c 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:11.431 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:11.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.420+0000 7eff4ed82640 1 -- 172.21.15.138:0/1316704889 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff481352f0 con 0x7eff481356f0 2024-09-11T05:22:11.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff4ed82640 1 -- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 msgr2=0x7eff4811e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:11.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff4ed82640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff4811e0c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7eff480a6e80 tx=0x7eff4001b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:11.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 client.?.objecter ms_handle_connect 0x7eff481356f0 2024-09-11T05:22:11.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:11.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 1 -- 172.21.15.138:0/1316704889 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff40009a10 con 0x7eff481356f0 2024-09-11T05:22:11.432 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: dump: 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: _finish_auth 0 2024-09-11T05:22:11.433 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 1 -- 172.21.15.138:0/1316704889 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff400be070 con 0x7eff481356f0 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 1 -- 172.21.15.138:0/1316704889 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff40009e30 con 0x7eff481356f0 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 5 monclient: authenticate success, global_id 84398 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 10 monclient: _renew_subs 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff4811fa90 con 0x7eff481356f0 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff4cd7e640 4 set_mon_vals no callback set 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 monclient: dump: 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:11.434 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 10 client.84398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 10 monclient: _renew_subs 2024-09-11T05:22:11.435 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff48121240 con 0x7eff481356f0 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 1 librados: init done 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff480a2dd0 con 0x7eff481356f0 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 1 -- 172.21.15.138:0/1316704889 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7eff400b8070 con 0x7eff481356f0 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 1 --2- 172.21.15.138:0/1316704889 >> v2:172.21.15.190:6800/1562782189 conn(0x7eff2c03ced0 0x7eff2c03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 1 -- 172.21.15.138:0/1316704889 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(600..600 src has 520..600) ==== 5929+0+0 (secure 0 0 0) 0x7eff400eaa40 con 0x7eff481356f0 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 10 client.84398.objecter ms_dispatch 0x7eff48000ba0 osd_map(600..600 src has 520..600) 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 3 client.84398.objecter handle_osd_map got epochs [600,600] > 0 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 3 client.84398.objecter handle_osd_map decoding full epoch 600 2024-09-11T05:22:11.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff3effd640 20 client.84398.objecter dump_active .. 0 homeless 2024-09-11T05:22:11.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff4e581640 1 --2- 172.21.15.138:0/1316704889 >> v2:172.21.15.190:6800/1562782189 conn(0x7eff2c03ced0 0x7eff2c03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:11.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.424+0000 7eff4e581640 10 monclient: get_auth_request con 0x7eff2c03ced0 auth_method 0 2024-09-11T05:22:11.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.428+0000 7eff3effd640 10 client.84398.objecter ms_handle_connect 0x7eff2c03ced0 2024-09-11T05:22:11.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.428+0000 7eff4e581640 1 --2- 172.21.15.138:0/1316704889 >> v2:172.21.15.190:6800/1562782189 conn(0x7eff2c03ced0 0x7eff2c03f390 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7eff48121d20 tx=0x7eff4400c150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:11.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.428+0000 7eff3effd640 1 -- 172.21.15.138:0/1316704889 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7eff400187c0 con 0x7eff481356f0 2024-09-11T05:22:11.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.428+0000 7eff3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:11.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.428+0000 7eff3effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:11.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.664+0000 7eff507e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:11.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.664+0000 7eff507e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:11.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.664+0000 7eff507e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:11.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.664+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff4809e0e0 con 0x7eff481356f0 2024-09-11T05:22:11.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.664+0000 7eff3effd640 1 -- 172.21.15.138:0/1316704889 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7eff4809e0e0 con 0x7eff481356f0 2024-09-11T05:22:11.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.664+0000 7eff3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:11.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.664+0000 7eff3effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:11.662 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 10 librados: watch_flush enter 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 10 librados: watch_flush exit 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 20 client.84398.objecter shutdown clearing up homeless session... 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 10 client.84398.objecter successfully canceled tick 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 >> v2:172.21.15.190:6800/1562782189 conn(0x7eff2c03ced0 msgr2=0x7eff2c03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 --2- 172.21.15.138:0/1316704889 >> v2:172.21.15.190:6800/1562782189 conn(0x7eff2c03ced0 0x7eff2c03f390 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7eff48121d20 tx=0x7eff4400c150 comp rx=0 tx=0).stop 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 10 monclient: shutdown 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 msgr2=0x7eff4811e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff4811e0c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7eff480a6e80 tx=0x7eff4001b820 comp rx=0 tx=0).stop 2024-09-11T05:22:11.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 shutdown_connections 2024-09-11T05:22:11.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 --2- 172.21.15.138:0/1316704889 >> v2:172.21.15.190:6800/1562782189 conn(0x7eff2c03ced0 0x7eff2c03f390 unknown :-1 s=CLOSED pgs=3472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7eff481356f0 0x7eff4811e0c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7eff4809e960 0x7eff481345b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 --2- 172.21.15.138:0/1316704889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff4806e0e0 0x7eff48134070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:11.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 >> 172.21.15.138:0/1316704889 conn(0x7eff48063640 msgr2=0x7eff480a5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:11.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 shutdown_connections 2024-09-11T05:22:11.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 -- 172.21.15.138:0/1316704889 wait complete. 2024-09-11T05:22:11.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:11.668+0000 7eff507e5640 1 librados: shutdown 2024-09-11T05:22:14.682 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient: get_monmap_and_config 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient: build_initial_monmap 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient: monmap: 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:14.815596+0000 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:14.815596+0000 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:14.810 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:14.811 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:14.811 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:14.811 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:14.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 1 Processor -- start 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 1 -- start start 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient: init 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf406e0e0 0x7fdcf4136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf406e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdcf4136170 0x7fdcf4136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf4136170 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcf40a1920 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf40a3270 con 0x7fdcf40a1920 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:14.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdcf406e4c0 con 0x7fdcf4136170 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf40a0af0 con 0x7fdcf406e0e0 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf9c27640 10 monclient(hunting): authenticate will time out at 1818.251745s 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.812+0000 7fdcf37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf406e0e0 0x7fdcf4136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf406e0e0 0x7fdcf4136720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55184/0 (socket says 172.21.15.138:55184) 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 1 -- 172.21.15.138:0/2497866905 learned_addr learned my addr 172.21.15.138:0/2497866905 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient(hunting): get_auth_request con 0x7fdcf406e0e0 auth_method 0 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:14.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf2ffd640 1 --2- 172.21.15.138:0/2497866905 >> v2:172.21.15.190:3300/0 conn(0x7fdcf4136170 0x7fdcf4136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf2ffd640 10 monclient(hunting): get_auth_request con 0x7fdcf4136170 auth_method 0 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient(hunting): handle_auth_done global_id 84401 payload 274 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 1 -- 172.21.15.138:0/2497866905 >> v2:172.21.15.190:3300/0 conn(0x7fdcf4136170 msgr2=0x7fdcf4136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 1 --2- 172.21.15.138:0/2497866905 >> v2:172.21.15.190:3300/0 conn(0x7fdcf4136170 0x7fdcf4136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 1 -- 172.21.15.138:0/2497866905 >> v1:172.21.15.190:6789/0 conn(0x7fdcf40a1920 legacy=0x7fdcf40a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 1 monclient: found mon.noname-b 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:14.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.816+0000 7fdcf37fe640 1 -- 172.21.15.138:0/2497866905 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf40a2f90 con 0x7fdcf406e0e0 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 -- 172.21.15.138:0/2497866905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf406e0e0 msgr2=0x7fdcf4136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 --2- 172.21.15.138:0/2497866905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf406e0e0 0x7fdcf4136720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009870 tx=0x7fdcdc01c240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dda09c414066e84 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 1 -- 172.21.15.138:0/2497866905 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc0b0070 con 0x7fdcf406e0e0 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: dump: 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:14.817 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: _finish_auth 0 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: _check_auth_tickets 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 1 -- 172.21.15.138:0/2497866905 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcdc0ab600 con 0x7fdcf406e0e0 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 1 -- 172.21.15.138:0/2497866905 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc0ab9d0 con 0x7fdcf406e0e0 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 5 monclient: authenticate success, global_id 84401 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 4 set_mon_vals no callback set 2024-09-11T05:22:14.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf27fc640 10 monclient: dump: 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient: shutdown 2024-09-11T05:22:14.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:14.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- 172.21.15.138:0/2497866905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf406e0e0 msgr2=0x7fdcf4136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:14.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/2497866905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf406e0e0 0x7fdcf4136720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009870 tx=0x7fdcdc01c240 comp rx=0 tx=0).stop 2024-09-11T05:22:14.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- 172.21.15.138:0/2497866905 shutdown_connections 2024-09-11T05:22:14.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/2497866905 >> v2:172.21.15.190:3300/0 conn(0x7fdcf4136170 0x7fdcf4136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:14.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/2497866905 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf406e0e0 0x7fdcf4136720 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:14.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- 172.21.15.138:0/2497866905 >> 172.21.15.138:0/2497866905 conn(0x7fdcf4063640 msgr2=0x7fdcf4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:14.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- 172.21.15.138:0/2497866905 shutdown_connections 2024-09-11T05:22:14.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- 172.21.15.138:0/2497866905 wait complete. 2024-09-11T05:22:14.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient: build_initial_monmap 2024-09-11T05:22:14.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient: monmap: 2024-09-11T05:22:14.828 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:14.828 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 librados: starting msgr at 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 librados: starting objecter 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 Processor -- start 2024-09-11T05:22:14.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- start start 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 librados: setting wanted keys 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 librados: calling monclient init 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient: init 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 0x7fdcf413d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf40a1920 0x7fdcf41d0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf40a1920 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcf4136170 0x7fdcf41d1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf4136170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:14.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf41410a0 con 0x7fdcf406e0e0 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf4140f20 con 0x7fdcf4136170 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf4141220 con 0x7fdcf40a1920 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf9c27640 10 monclient(hunting): authenticate will time out at 1818.258872s 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf40a1920 0x7fdcf41d0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 0x7fdcf413d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 0x7fdcf413d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:37426/0 (socket says 172.21.15.138:37426) 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf40a1920 0x7fdcf41d0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55200/0 (socket says 172.21.15.138:55200) 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 -- 172.21.15.138:0/89388599 learned_addr learned my addr 172.21.15.138:0/89388599 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:14.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 10 monclient(hunting): get_auth_request con 0x7fdcf406e0e0 auth_method 0 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient(hunting): get_auth_request con 0x7fdcf40a1920 auth_method 0 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 -- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 msgr2=0x7fdcf413d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 -- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 msgr2=0x7fdcf413d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 0x7fdcf413d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:14.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf37fe640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 0x7fdcf413d0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf3fff640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcf4136170 0x7fdcf41d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf3fff640 10 monclient(hunting): get_auth_request con 0x7fdcf4136170 auth_method 0 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient(hunting): handle_auth_done global_id 84404 payload 824 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 1 -- 172.21.15.138:0/89388599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcf4136170 msgr2=0x7fdcf41d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcf4136170 0x7fdcf41d1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 1 -- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 msgr2=0x7fdcf413d0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 0x7fdcf413d0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:14.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 1 monclient: found mon.noname-c 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.820+0000 7fdcf2ffd640 1 -- 172.21.15.138:0/89388599 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf413c960 con 0x7fdcf40a1920 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf2ffd640 1 -- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf40a1920 msgr2=0x7fdcf41d0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf2ffd640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf40a1920 0x7fdcf41d0b10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdce400d5a0 tx=0x7fdce400da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 client.?.objecter ms_handle_connect 0x7fdcf40a1920 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 1 -- 172.21.15.138:0/89388599 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce4019070 con 0x7fdcf40a1920 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: dump: 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:14.834 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: _finish_auth 0 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: _check_auth_tickets 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:14.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 1 -- 172.21.15.138:0/89388599 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce4004450 con 0x7fdcf40a1920 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 1 -- 172.21.15.138:0/89388599 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce4003860 con 0x7fdcf40a1920 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 monclient: dump: 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:14.836 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf1ffb640 4 set_mon_vals no callback set 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 5 monclient: authenticate success, global_id 84404 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 10 monclient: _renew_subs 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf4133890 con 0x7fdcf40a1920 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 10 client.84404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 10 monclient: _renew_subs 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:14.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf413c680 con 0x7fdcf40a1920 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 1 librados: init done 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf40a1db0 con 0x7fdcf40a1920 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 1 -- 172.21.15.138:0/89388599 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdce4013070 con 0x7fdcf40a1920 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 1 --2- 172.21.15.138:0/89388599 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcd403cef0 0x7fdcd403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 1 -- 172.21.15.138:0/89388599 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(603..603 src has 520..603) ==== 5929+0+0 (secure 0 0 0) 0x7fdce404f4d0 con 0x7fdcf40a1920 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 10 client.84404.objecter ms_dispatch 0x7fdcf4000ba0 osd_map(603..603 src has 520..603) 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 3 client.84404.objecter handle_osd_map got epochs [603,603] > 0 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.824+0000 7fdcebfff640 3 client.84404.objecter handle_osd_map decoding full epoch 603 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.828+0000 7fdcebfff640 20 client.84404.objecter dump_active .. 0 homeless 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.828+0000 7fdcf37fe640 1 --2- 172.21.15.138:0/89388599 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcd403cef0 0x7fdcd403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:14.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.828+0000 7fdcf37fe640 10 monclient: get_auth_request con 0x7fdcd403cef0 auth_method 0 2024-09-11T05:22:14.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.828+0000 7fdcf37fe640 1 --2- 172.21.15.138:0/89388599 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcd403cef0 0x7fdcd403f3b0 secure :-1 s=READY pgs=3473 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc000c10 tx=0x7fdcdc0a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:14.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.828+0000 7fdcebfff640 10 client.84404.objecter ms_handle_connect 0x7fdcd403cef0 2024-09-11T05:22:14.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.832+0000 7fdcebfff640 1 -- 172.21.15.138:0/89388599 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fdce4012510 con 0x7fdcf40a1920 2024-09-11T05:22:14.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.832+0000 7fdcebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:14.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:14.832+0000 7fdcebfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:15.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.080+0000 7fdcf9c27640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:15.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.080+0000 7fdcf9c27640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:15.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.080+0000 7fdcf9c27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:15.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.080+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdcf41359d0 con 0x7fdcf40a1920 2024-09-11T05:22:15.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcebfff640 1 -- 172.21.15.138:0/89388599 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fdce4011380 con 0x7fdcf40a1920 2024-09-11T05:22:15.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:15.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcebfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:15.080 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":97,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:15.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 10 librados: watch_flush enter 2024-09-11T05:22:15.082 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 10 librados: watch_flush exit 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 20 client.84404.objecter shutdown clearing up homeless session... 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 10 client.84404.objecter successfully canceled tick 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcd403cef0 msgr2=0x7fdcd403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/89388599 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcd403cef0 0x7fdcd403f3b0 secure :-1 s=READY pgs=3473 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc000c10 tx=0x7fdcdc0a6000 comp rx=0 tx=0).stop 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 10 monclient: shutdown 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf40a1920 msgr2=0x7fdcf41d0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf40a1920 0x7fdcf41d0b10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdce400d5a0 tx=0x7fdce400da60 comp rx=0 tx=0).stop 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 shutdown_connections 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/89388599 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcd403cef0 0x7fdcd403f3b0 unknown :-1 s=CLOSED pgs=3473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcf4136170 0x7fdcf41d1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdcf40a1920 0x7fdcf41d0b10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 --2- 172.21.15.138:0/89388599 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdcf406e0e0 0x7fdcf413d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:15.083 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 >> 172.21.15.138:0/89388599 conn(0x7fdcf4063640 msgr2=0x7fdcf413d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:15.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 shutdown_connections 2024-09-11T05:22:15.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 -- 172.21.15.138:0/89388599 wait complete. 2024-09-11T05:22:15.084 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:15.084+0000 7fdcf9c27640 1 librados: shutdown 2024-09-11T05:22:18.099 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:18.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient: get_monmap_and_config 2024-09-11T05:22:18.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient: build_initial_monmap 2024-09-11T05:22:18.225 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient: monmap: 2024-09-11T05:22:18.225 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:18.231182+0000 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:18.231182+0000 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:18.226 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 1 Processor -- start 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 1 -- start start 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient: init 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f88881388c0 0x7f8888138ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient(hunting): picked mon.noname-c con 0x7f88881388c0 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8888074240 0x7f88880a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient(hunting): picked mon.noname-b con 0x7f8888074240 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88880a1360 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88881391e0 con 0x7f88880a1360 2024-09-11T05:22:18.229 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f88880a61a0 con 0x7f88881388c0 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f88880a5c40 con 0x7f8888074240 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.228+0000 7f888f38a640 10 monclient(hunting): authenticate will time out at 1821.668000s 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8888074240 0x7f88880a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8888074240 0x7f88880a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48402/0 (socket says 172.21.15.138:48402) 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 1 -- 172.21.15.138:0/3191787978 learned_addr learned my addr 172.21.15.138:0/3191787978 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient(hunting): get_auth_request con 0x7f8888074240 auth_method 0 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:18.230 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888d126640 1 --2- 172.21.15.138:0/3191787978 >> v2:172.21.15.190:3300/0 conn(0x7f88881388c0 0x7f8888138ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888d126640 10 monclient(hunting): get_auth_request con 0x7f88881388c0 auth_method 0 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888d126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888d126640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888d126640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient(hunting): handle_auth_done global_id 84407 payload 274 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 1 -- 172.21.15.138:0/3191787978 >> v2:172.21.15.190:3300/0 conn(0x7f88881388c0 msgr2=0x7f8888138ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 1 --2- 172.21.15.138:0/3191787978 >> v2:172.21.15.190:3300/0 conn(0x7f88881388c0 0x7f8888138ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 1 -- 172.21.15.138:0/3191787978 >> v1:172.21.15.190:6789/0 conn(0x7f88880a1360 legacy=0x7f888813f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient: _finish_hunting 0 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 1 monclient: found mon.noname-b 2024-09-11T05:22:18.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:18.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:18.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.232+0000 7f888c925640 1 -- 172.21.15.138:0/3191787978 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88880a7240 con 0x7f8888074240 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 -- 172.21.15.138:0/3191787978 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8888074240 msgr2=0x7f88880a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 --2- 172.21.15.138:0/3191787978 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8888074240 0x7f88880a0e20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8878009870 tx=0x7f887801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6cf33aebf59f43c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 1 -- 172.21.15.138:0/3191787978 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88780b1070 con 0x7f8888074240 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: dump: 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:18.234 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: _finish_auth 0 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 1 -- 172.21.15.138:0/3191787978 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88780ac620 con 0x7f8888074240 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 5 monclient: authenticate success, global_id 84407 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 1 -- 172.21.15.138:0/3191787978 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88780ac9f0 con 0x7f8888074240 2024-09-11T05:22:18.235 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 4 set_mon_vals no callback set 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f887ffff640 10 monclient: dump: 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient: shutdown 2024-09-11T05:22:18.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- 172.21.15.138:0/3191787978 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8888074240 msgr2=0x7f88880a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 --2- 172.21.15.138:0/3191787978 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8888074240 0x7f88880a0e20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8878009870 tx=0x7f887801c420 comp rx=0 tx=0).stop 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- 172.21.15.138:0/3191787978 shutdown_connections 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 --2- 172.21.15.138:0/3191787978 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8888074240 0x7f88880a0e20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 --2- 172.21.15.138:0/3191787978 >> v2:172.21.15.190:3300/0 conn(0x7f88881388c0 0x7f8888138ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- 172.21.15.138:0/3191787978 >> 172.21.15.138:0/3191787978 conn(0x7f888806cc20 msgr2=0x7f888806d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- 172.21.15.138:0/3191787978 shutdown_connections 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- 172.21.15.138:0/3191787978 wait complete. 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient: build_initial_monmap 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient: monmap: 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:18.237 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 librados: starting msgr at 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 librados: starting objecter 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 Processor -- start 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- start start 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 librados: setting wanted keys 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 librados: calling monclient init 2024-09-11T05:22:18.238 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient: init 2024-09-11T05:22:18.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:18.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:22:18.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 0x7f88881d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:18.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient(hunting): picked mon.noname-a con 0x7f88880a1360 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:18.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88881388c0 0x7f88881d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:18.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient(hunting): picked mon.noname-c con 0x7f88881388c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:18.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88881d1430 0x7f8888132010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient(hunting): picked mon.noname-b con 0x7f88881d1430 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8888070750 con 0x7f88880a1360 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8888143560 con 0x7f88881d1430 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f88881436e0 con 0x7f88881388c0 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888f38a640 10 monclient(hunting): authenticate will time out at 1821.675164s 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 0x7f88881d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88881388c0 0x7f88881d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 0x7f88881d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45514/0 (socket says 172.21.15.138:45514) 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 1 -- 172.21.15.138:0/3065875990 learned_addr learned my addr 172.21.15.138:0/3065875990 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient(hunting): get_auth_request con 0x7f88881388c0 auth_method 0 2024-09-11T05:22:18.243 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 10 monclient(hunting): get_auth_request con 0x7f88880a1360 auth_method 0 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 1 -- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 msgr2=0x7f88881d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 1 -- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 msgr2=0x7f88881d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 0x7f88881d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d126640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 0x7f88881d09b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:18.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient(hunting): handle_auth_done global_id 84410 payload 824 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888d927640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88881d1430 0x7f8888132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 -- 172.21.15.138:0/3065875990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88881d1430 msgr2=0x7f8888132010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88881d1430 0x7f8888132010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 -- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 msgr2=0x7f88881d09b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 0x7f88881d09b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient: _finish_hunting 0 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 monclient: found mon.noname-c 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.236+0000 7f888c925640 1 -- 172.21.15.138:0/3065875990 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88881364f0 con 0x7f88881388c0 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888c925640 1 -- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88881388c0 msgr2=0x7f88881d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888c925640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88881388c0 0x7f88881d0ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f88780ae000 tx=0x7f88780191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 client.?.objecter ms_handle_connect 0x7f88881388c0 2024-09-11T05:22:18.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 1 -- 172.21.15.138:0/3065875990 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88780b1040 con 0x7f88881388c0 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: dump: 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: _finish_auth 0 2024-09-11T05:22:18.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: _check_auth_tickets 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 1 -- 172.21.15.138:0/3065875990 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88780ac420 con 0x7f88881388c0 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 5 monclient: authenticate success, global_id 84410 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 10 monclient: _renew_subs 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f888813cc50 con 0x7f88881388c0 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 1 -- 172.21.15.138:0/3065875990 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88780b5660 con 0x7f88881388c0 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 monclient: dump: 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:18.247 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 10 client.84410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 10 monclient: _renew_subs 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887f7fe640 4 set_mon_vals no callback set 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f888813d7b0 con 0x7f88881388c0 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 1 librados: init done 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:18.248 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88880a0e20 con 0x7f88881388c0 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 1 -- 172.21.15.138:0/3065875990 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f88780be410 con 0x7f88881388c0 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 1 --2- 172.21.15.138:0/3065875990 >> v2:172.21.15.190:6800/1562782189 conn(0x7f886c03cef0 0x7f886c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 1 -- 172.21.15.138:0/3065875990 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(607..607 src has 520..607) ==== 6041+0+0 (secure 0 0 0) 0x7f88780e9230 con 0x7f88881388c0 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 10 client.84410.objecter ms_dispatch 0x7f8888000ba0 osd_map(607..607 src has 520..607) 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 3 client.84410.objecter handle_osd_map got epochs [607,607] > 0 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 3 client.84410.objecter handle_osd_map decoding full epoch 607 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f887dffb640 20 client.84410.objecter dump_active .. 0 homeless 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.240+0000 7f888d126640 1 --2- 172.21.15.138:0/3065875990 >> v2:172.21.15.190:6800/1562782189 conn(0x7f886c03cef0 0x7f886c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.244+0000 7f888d126640 10 monclient: get_auth_request con 0x7f886c03cef0 auth_method 0 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.244+0000 7f887dffb640 10 client.84410.objecter ms_handle_connect 0x7f886c03cef0 2024-09-11T05:22:18.249 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.244+0000 7f888d126640 1 --2- 172.21.15.138:0/3065875990 >> v2:172.21.15.190:6800/1562782189 conn(0x7f886c03cef0 0x7f886c03f3b0 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 crypto rx=0x7f8884003fd0 tx=0x7f88840039a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.248+0000 7f887dffb640 1 -- 172.21.15.138:0/3065875990 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f88780c3020 con 0x7f88881388c0 2024-09-11T05:22:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.248+0000 7f887dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:18.250 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.248+0000 7f887dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:18.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.476+0000 7f888f38a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:18.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:18.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:18.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f888813d4c0 con 0x7f88881388c0 2024-09-11T05:22:18.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f887dffb640 1 -- 172.21.15.138:0/3065875990 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f8878016130 con 0x7f88881388c0 2024-09-11T05:22:18.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f887dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:18.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f887dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:18.477 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 10 librados: watch_flush enter 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 10 librados: watch_flush exit 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 20 client.84410.objecter shutdown clearing up homeless session... 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 10 client.84410.objecter successfully canceled tick 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 >> v2:172.21.15.190:6800/1562782189 conn(0x7f886c03cef0 msgr2=0x7f886c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 --2- 172.21.15.138:0/3065875990 >> v2:172.21.15.190:6800/1562782189 conn(0x7f886c03cef0 0x7f886c03f3b0 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 crypto rx=0x7f8884003fd0 tx=0x7f88840039a0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 10 monclient: shutdown 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88881388c0 msgr2=0x7f88881d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88881388c0 0x7f88881d0ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f88780ae000 tx=0x7f88780191c0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 shutdown_connections 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 --2- 172.21.15.138:0/3065875990 >> v2:172.21.15.190:6800/1562782189 conn(0x7f886c03cef0 0x7f886c03f3b0 unknown :-1 s=CLOSED pgs=3474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88881d1430 0x7f8888132010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f88881388c0 0x7f88881d0ef0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 --2- 172.21.15.138:0/3065875990 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f88880a1360 0x7f88881d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:18.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 >> 172.21.15.138:0/3065875990 conn(0x7f888806cc20 msgr2=0x7f8888072a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:18.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 shutdown_connections 2024-09-11T05:22:18.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 -- 172.21.15.138:0/3065875990 wait complete. 2024-09-11T05:22:18.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:18.480+0000 7f888f38a640 1 librados: shutdown 2024-09-11T05:22:21.495 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:21.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient: get_monmap_and_config 2024-09-11T05:22:21.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient: build_initial_monmap 2024-09-11T05:22:21.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient: monmap: 2024-09-11T05:22:21.629 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:21.629 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:21.629 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:21.634706+0000 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:21.634706+0000 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:21.630 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:21.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 1 Processor -- start 2024-09-11T05:22:21.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 1 -- start start 2024-09-11T05:22:21.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient: init 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c06e0e0 0x7f440c09e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient(hunting): picked mon.noname-b con 0x7f440c06e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f440c09e960 0x7f440c09ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient(hunting): picked mon.noname-a con 0x7f440c09e960 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f440c1356f0 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f440c09f540 con 0x7f440c1356f0 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f440c09f280 con 0x7f440c09e960 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:21.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f440c06e4c0 con 0x7f440c06e0e0 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f44138f9640 10 monclient(hunting): authenticate will time out at 1825.071572s 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c06e0e0 0x7f440c09e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f440c09e960 0x7f440c09ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c06e0e0 0x7f440c09e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48452/0 (socket says 172.21.15.138:48452) 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 1 -- 172.21.15.138:0/3713910090 learned_addr learned my addr 172.21.15.138:0/3713910090 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411e96640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f440c1356f0 0x7f440c135b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45616/0 (socket says 172.21.15.138:45616) 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 10 monclient(hunting): get_auth_request con 0x7f440c06e0e0 auth_method 0 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 1 --2- 172.21.15.138:0/3713910090 >> v2:172.21.15.138:3300/0 conn(0x7f440c09e960 0x7f440c09ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45536/0 (socket says 172.21.15.138:45536) 2024-09-11T05:22:21.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 10 monclient(hunting): get_auth_request con 0x7f440c09e960 auth_method 0 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4410e94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.632+0000 7f4411695640 10 monclient(hunting): handle_auth_done global_id 84413 payload 274 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4411695640 1 -- 172.21.15.138:0/3713910090 >> v2:172.21.15.138:3300/0 conn(0x7f440c09e960 msgr2=0x7f440c09ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4411695640 1 --2- 172.21.15.138:0/3713910090 >> v2:172.21.15.138:3300/0 conn(0x7f440c09e960 0x7f440c09ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4411695640 1 -- 172.21.15.138:0/3713910090 >> v1:172.21.15.138:6789/0 conn(0x7f440c1356f0 legacy=0x7f440c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4411695640 10 monclient: _finish_hunting 0 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4411695640 1 monclient: found mon.noname-b 2024-09-11T05:22:21.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4411695640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:21.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4411695640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:21.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4411695640 1 -- 172.21.15.138:0/3713910090 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c09fb00 con 0x7f440c06e0e0 2024-09-11T05:22:21.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.636+0000 7f4410e94640 1 --2- 172.21.15.138:0/3713910090 >> v2:172.21.15.138:3300/0 conn(0x7f440c09e960 0x7f440c09ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:21.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 -- 172.21.15.138:0/3713910090 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c06e0e0 msgr2=0x7f440c09e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:21.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 --2- 172.21.15.138:0/3713910090 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c06e0e0 0x7f440c09e420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4408009870 tx=0x7f440801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eef4b8f64d676200 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 1 -- 172.21.15.138:0/3713910090 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080b0070 con 0x7f440c06e0e0 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: dump: 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:21.640 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: _finish_auth 0 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: _check_auth_tickets 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 1 -- 172.21.15.138:0/3713910090 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44080a4e50 con 0x7f440c06e0e0 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 1 -- 172.21.15.138:0/3713910090 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080ab820 con 0x7f440c06e0e0 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4403fff640 10 monclient: dump: 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:21.641 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 5 monclient: authenticate success, global_id 84413 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:21.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 4 set_mon_vals no callback set 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient: shutdown 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- 172.21.15.138:0/3713910090 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c06e0e0 msgr2=0x7f440c09e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 --2- 172.21.15.138:0/3713910090 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c06e0e0 0x7f440c09e420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4408009870 tx=0x7f440801c4b0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- 172.21.15.138:0/3713910090 shutdown_connections 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 --2- 172.21.15.138:0/3713910090 >> v2:172.21.15.138:3300/0 conn(0x7f440c09e960 0x7f440c09ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 --2- 172.21.15.138:0/3713910090 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c06e0e0 0x7f440c09e420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- 172.21.15.138:0/3713910090 >> 172.21.15.138:0/3713910090 conn(0x7f440c063640 msgr2=0x7f440c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- 172.21.15.138:0/3713910090 shutdown_connections 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- 172.21.15.138:0/3713910090 wait complete. 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient: build_initial_monmap 2024-09-11T05:22:21.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient: monmap: 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 librados: starting msgr at 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 librados: starting objecter 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 Processor -- start 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- start start 2024-09-11T05:22:21.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 librados: setting wanted keys 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 librados: calling monclient init 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient: init 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f440c06e0e0 0x7f440c1d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient(hunting): picked mon.noname-a con 0x7f440c06e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c09e960 0x7f440c134940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient(hunting): picked mon.noname-c con 0x7f440c09e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f440c1356f0 0x7f440c13b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:21.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient(hunting): picked mon.noname-b con 0x7f440c1356f0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0a2b40 con 0x7f440c06e0e0 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0a29c0 con 0x7f440c1356f0 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f440c0a2cc0 con 0x7f440c09e960 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f44138f9640 10 monclient(hunting): authenticate will time out at 1825.078986s 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f440c06e0e0 0x7f440c1d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f440c06e0e0 0x7f440c1d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45538/0 (socket says 172.21.15.138:45538) 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 -- 172.21.15.138:0/1817149789 learned_addr learned my addr 172.21.15.138:0/1817149789 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:21.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient(hunting): get_auth_request con 0x7f440c06e0e0 auth_method 0 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4410e94640 1 --2- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c09e960 0x7f440c134940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4410e94640 10 monclient(hunting): get_auth_request con 0x7f440c09e960 auth_method 0 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4410e94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4410e94640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4410e94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:21.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4410e94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4410e94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4410e94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient(hunting): handle_auth_done global_id 114102 payload 824 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 -- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c09e960 msgr2=0x7f440c134940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 --2- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c09e960 0x7f440c134940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 -- 172.21.15.138:0/1817149789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f440c1356f0 msgr2=0x7f440c13b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 --2- 172.21.15.138:0/1817149789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f440c1356f0 0x7f440c13b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient: _finish_hunting 0 2024-09-11T05:22:21.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 monclient: found mon.noname-a 2024-09-11T05:22:21.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:21.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:21.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.640+0000 7f4411695640 1 -- 172.21.15.138:0/1817149789 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c1352d0 con 0x7f440c06e0e0 2024-09-11T05:22:21.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4411695640 1 -- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f440c06e0e0 msgr2=0x7f440c1d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4411695640 1 --2- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f440c06e0e0 0x7f440c1d09b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44080ae040 tx=0x7f4408019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 client.?.objecter ms_handle_connect 0x7f440c06e0e0 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 1 -- 172.21.15.138:0/1817149789 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080b9070 con 0x7f440c06e0e0 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: dump: 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:21.653 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: _finish_auth 0 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:21.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 5 monclient: authenticate success, global_id 114102 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 1 -- 172.21.15.138:0/1817149789 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44080b4420 con 0x7f440c06e0e0 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 1 -- 172.21.15.138:0/1817149789 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080b0040 con 0x7f440c06e0e0 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 monclient: dump: 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:21.655 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44037fe640 4 set_mon_vals no callback set 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 10 monclient: _renew_subs 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c123030 con 0x7f440c06e0e0 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 10 client.114102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 10 monclient: _renew_subs 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:21.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c09e110 con 0x7f440c06e0e0 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 1 librados: init done 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c09e420 con 0x7f440c06e0e0 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 1 -- 172.21.15.138:0/1817149789 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f44080bd3b0 con 0x7f440c06e0e0 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 1 --2- 172.21.15.138:0/1817149789 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43f003cef0 0x7f43f003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 1 -- 172.21.15.138:0/1817149789 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(610..610 src has 520..610) ==== 5604+0+0 (secure 0 0 0) 0x7f44080e8e70 con 0x7f440c06e0e0 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 10 client.114102.objecter ms_dispatch 0x7f440c000ba0 osd_map(610..610 src has 520..610) 2024-09-11T05:22:21.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 3 client.114102.objecter handle_osd_map got epochs [610,610] > 0 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 3 client.114102.objecter handle_osd_map decoding full epoch 610 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4401ffb640 20 client.114102.objecter dump_active .. 0 homeless 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4410e94640 1 --2- 172.21.15.138:0/1817149789 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43f003cef0 0x7f43f003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.644+0000 7f4410e94640 10 monclient: get_auth_request con 0x7f43f003cef0 auth_method 0 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.648+0000 7f4410e94640 1 --2- 172.21.15.138:0/1817149789 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43f003cef0 0x7f43f003f3b0 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00b820 tx=0x7f43fc00b3a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.648+0000 7f4401ffb640 10 client.114102.objecter ms_handle_connect 0x7f43f003cef0 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.652+0000 7f4401ffb640 1 -- 172.21.15.138:0/1817149789 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f4408016130 con 0x7f440c06e0e0 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.652+0000 7f4401ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:21.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.652+0000 7f4401ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:21.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.888+0000 7f44138f9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:21.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.888+0000 7f44138f9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:21.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.888+0000 7f44138f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:21.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.888+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f440c121fd0 con 0x7f440c06e0e0 2024-09-11T05:22:21.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.888+0000 7f4401ffb640 1 -- 172.21.15.138:0/1817149789 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f44080a4df0 con 0x7f440c06e0e0 2024-09-11T05:22:21.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.888+0000 7f4401ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:21.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.888+0000 7f4401ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:21.887 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 10 librados: watch_flush enter 2024-09-11T05:22:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 10 librados: watch_flush exit 2024-09-11T05:22:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 20 client.114102.objecter shutdown clearing up homeless session... 2024-09-11T05:22:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 10 client.114102.objecter successfully canceled tick 2024-09-11T05:22:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43f003cef0 msgr2=0x7f43f003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 --2- 172.21.15.138:0/1817149789 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43f003cef0 0x7f43f003f3b0 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00b820 tx=0x7f43fc00b3a0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 10 monclient: shutdown 2024-09-11T05:22:21.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f440c06e0e0 msgr2=0x7f440c1d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 --2- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f440c06e0e0 0x7f440c1d09b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44080ae040 tx=0x7f4408019250 comp rx=0 tx=0).stop 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 shutdown_connections 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 --2- 172.21.15.138:0/1817149789 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43f003cef0 0x7f43f003f3b0 unknown :-1 s=CLOSED pgs=3476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 --2- 172.21.15.138:0/1817149789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f440c1356f0 0x7f440c13b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 --2- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f440c09e960 0x7f440c134940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 --2- 172.21.15.138:0/1817149789 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f440c06e0e0 0x7f440c1d09b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 >> 172.21.15.138:0/1817149789 conn(0x7f440c063640 msgr2=0x7f440c0a5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 shutdown_connections 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 -- 172.21.15.138:0/1817149789 wait complete. 2024-09-11T05:22:21.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:21.892+0000 7f44138f9640 1 librados: shutdown 2024-09-11T05:22:21.906 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:22:21.906 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:22:21.906 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:22:22.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient: get_monmap_and_config 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient: build_initial_monmap 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient: monmap: 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:22.037562+0000 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:22.037562+0000 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:22.032 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:22.033 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 1 Processor -- start 2024-09-11T05:22:22.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 1 -- start start 2024-09-11T05:22:22.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient: init 2024-09-11T05:22:22.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:22.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4880070010 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488009e6e0 0x7f488009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient(hunting): picked mon.noname-b con 0x7f488009e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f488013d640 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4880135b60 con 0x7f4880070010 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48800a8990 con 0x7f488013d640 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f48800703f0 con 0x7f488009e6e0 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f4885df2640 10 monclient(hunting): authenticate will time out at 1825.473692s 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488009e6e0 0x7f488009eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487f7fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f4880070010 0x7f4880135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45620/0 (socket says 172.21.15.138:45620) 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487f7fe640 1 -- 172.21.15.138:0/3963588149 learned_addr learned my addr 172.21.15.138:0/3963588149 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:22:22.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient(hunting): get_auth_request con 0x7f488009e6e0 auth_method 0 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient(hunting): handle_auth_done global_id 84419 payload 274 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 1 -- 172.21.15.138:0/3963588149 >> v1:172.21.15.190:6789/0 conn(0x7f488013d640 legacy=0x7f488013fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 1 -- 172.21.15.138:0/3963588149 >> v1:172.21.15.138:6789/0 conn(0x7f4880070010 legacy=0x7f4880135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient: _finish_hunting 0 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 1 monclient: found mon.noname-b 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:22.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.036+0000 7f487effd640 1 -- 172.21.15.138:0/3963588149 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488009cc40 con 0x7f488009e6e0 2024-09-11T05:22:22.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487effd640 1 -- 172.21.15.138:0/3963588149 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488009e6e0 msgr2=0x7f488009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487effd640 1 --2- 172.21.15.138:0/3963588149 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488009e6e0 0x7f488009eae0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f48680097f0 tx=0x7f486801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97cf080452f50610 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 1 -- 172.21.15.138:0/3963588149 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48680b0070 con 0x7f488009e6e0 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: dump: 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.039 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 5 monclient: authenticate success, global_id 84419 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 1 -- 172.21.15.138:0/3963588149 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48680ab600 con 0x7f488009e6e0 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 1 -- 172.21.15.138:0/3963588149 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48680a4e60 con 0x7f488009e6e0 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 4 set_mon_vals no callback set 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f487e7fc640 10 monclient: dump: 2024-09-11T05:22:22.040 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.041 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 10 monclient: shutdown 2024-09-11T05:22:22.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:22.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 1 -- 172.21.15.138:0/3963588149 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488009e6e0 msgr2=0x7f488009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 1 --2- 172.21.15.138:0/3963588149 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488009e6e0 0x7f488009eae0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f48680097f0 tx=0x7f486801c3a0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 1 -- 172.21.15.138:0/3963588149 shutdown_connections 2024-09-11T05:22:22.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 1 --2- 172.21.15.138:0/3963588149 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488009e6e0 0x7f488009eae0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.040+0000 7f4885df2640 1 -- 172.21.15.138:0/3963588149 >> 172.21.15.138:0/3963588149 conn(0x7f48800ab830 msgr2=0x7f48800a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 -- 172.21.15.138:0/3963588149 shutdown_connections 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 -- 172.21.15.138:0/3963588149 wait complete. 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient: build_initial_monmap 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient: monmap: 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:22.049 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 librados: starting msgr at 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 librados: starting objecter 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 Processor -- start 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 -- start start 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 librados: setting wanted keys 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 librados: calling monclient init 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient: init 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4880070010 0x7f4880132040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient(hunting): picked mon.noname-b con 0x7f4880070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f488009e6e0 0x7f4880136260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient(hunting): picked mon.noname-a con 0x7f488009e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488013d640 0x7f48801367a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient(hunting): picked mon.noname-c con 0x7f488013d640 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4880143e10 con 0x7f488009e6e0 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4880143c90 con 0x7f4880070010 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4880143f90 con 0x7f488013d640 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f4885df2640 10 monclient(hunting): authenticate will time out at 1825.481523s 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f488009e6e0 0x7f4880136260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488013d640 0x7f48801367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f488009e6e0 0x7f4880136260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45552/0 (socket says 172.21.15.138:45552) 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 1 -- 172.21.15.138:0/84783805 learned_addr learned my addr 172.21.15.138:0/84783805 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 10 monclient(hunting): get_auth_request con 0x7f488009e6e0 auth_method 0 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient(hunting): get_auth_request con 0x7f488013d640 auth_method 0 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487f7fe640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4880070010 0x7f4880132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487f7fe640 10 monclient(hunting): get_auth_request con 0x7f4880070010 auth_method 0 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient(hunting): handle_auth_done global_id 84422 payload 824 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4880070010 msgr2=0x7f4880132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4880070010 0x7f4880132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f488009e6e0 msgr2=0x7f4880136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f488009e6e0 0x7f4880136260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 1 monclient: found mon.noname-c 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487ffff640 1 -- 172.21.15.138:0/84783805 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4880137ba0 con 0x7f488013d640 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.044+0000 7f487effd640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f488009e6e0 0x7f4880136260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487ffff640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488013d640 msgr2=0x7f48801367a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487ffff640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488013d640 0x7f48801367a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f487000c3e0 tx=0x7f487000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 client.?.objecter ms_handle_connect 0x7f488013d640 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48700119a0 con 0x7f488013d640 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: dump: 2024-09-11T05:22:22.054 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: _finish_auth 0 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: _check_auth_tickets 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 5 monclient: authenticate success, global_id 84422 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4870017070 con 0x7f488013d640 2024-09-11T05:22:22.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 10 monclient: _renew_subs 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48800a2330 con 0x7f488013d640 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870011d80 con 0x7f488013d640 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 monclient: dump: 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.056 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487dffb640 4 set_mon_vals no callback set 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 10 client.84422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 10 monclient: _renew_subs 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4880132580 con 0x7f488013d640 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 1 librados: init done 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48800a3eb0 0x7f4880136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f48800a3430 con 0x7f48800a3eb0 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48800a3eb0 0x7f4880136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 10 monclient: get_auth_request con 0x7f48800a3eb0 auth_method 0 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 10 monclient(hunting): handle_auth_done global_id 114108 payload 824 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 1 -- 172.21.15.138:0/84783805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f488009eae0 con 0x7f48800a3eb0 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f487001c460 con 0x7f488013d640 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 1 --2- 172.21.15.138:0/84783805 >> v2:172.21.15.190:6800/1562782189 conn(0x7f486003cef0 0x7f486003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(611..611 src has 520..611) ==== 5977+0+0 (secure 0 0 0) 0x7f487004f8c0 con 0x7f488013d640 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 client.84422.objecter ms_dispatch 0x7f4880000ba0 osd_map(611..611 src has 520..611) 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 3 client.84422.objecter handle_osd_map got epochs [611,611] > 0 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 3 client.84422.objecter handle_osd_map decoding full epoch 611 2024-09-11T05:22:22.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 20 client.84422.objecter dump_active .. 0 homeless 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487effd640 1 --2- 172.21.15.138:0/84783805 >> v2:172.21.15.190:6800/1562782189 conn(0x7f486003cef0 0x7f486003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487effd640 10 monclient: get_auth_request con 0x7f486003cef0 auth_method 0 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 client.84422.objecter ms_handle_connect 0x7f48800a3eb0 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48800a3eb0 msgr2=0x7f4880136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f4877fff640 10 client.84422.objecter resend_mon_ops 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.048+0000 7f487f7fe640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48800a3eb0 0x7f4880136ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f487800ebc0 tx=0x7f487800c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878017020 con 0x7f48800a3eb0 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 10 monclient: dump: 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.059 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f487800cdf0 con 0x7f48800a3eb0 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48800a3eb0 msgr2=0x7f4880136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48800a3eb0 0x7f4880136ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f487800ebc0 tx=0x7f487800c4b0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48800a3eb0 msgr2=0x7f4880136ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:22.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f48800a3eb0 0x7f4880136ce0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f487800ebc0 tx=0x7f487800c4b0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f4877fff640 10 client.84422.objecter ms_handle_connect 0x7f486003cef0 2024-09-11T05:22:22.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.052+0000 7f487effd640 1 --2- 172.21.15.138:0/84783805 >> v2:172.21.15.190:6800/1562782189 conn(0x7f486003cef0 0x7f486003f3b0 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 crypto rx=0x7f4868009740 tx=0x7f48680a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f4885df2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:22:22.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f4885df2640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4880137220 0x7f48800a9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f4885df2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4880137600 con 0x7f4880137220 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4880137220 0x7f48800a9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 10 monclient: get_auth_request con 0x7f4880137220 auth_method 0 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 10 monclient(hunting): handle_auth_done global_id 114111 payload 824 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.056+0000 7f487f7fe640 1 -- 172.21.15.138:0/84783805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48801327b0 con 0x7f4880137220 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 10 client.84422.objecter ms_handle_connect 0x7f4880137220 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 10 client.84422.objecter resend_mon_ops 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f487f7fe640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4880137220 msgr2=0x7f48800a9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f487f7fe640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4880137220 0x7f48800a9570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4878018000 tx=0x7f4878004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878017040 con 0x7f4880137220 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 10 monclient: dump: 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f487801e220 con 0x7f4880137220 2024-09-11T05:22:22.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4880137220 msgr2=0x7f48800a9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4880137220 0x7f48800a9570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4878018000 tx=0x7f4878004cd0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4880137220 msgr2=0x7f48800a9570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4877fff640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4880137220 0x7f48800a9570 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4878018000 tx=0x7f4878004cd0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:22.065 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 58, 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 14099, 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.066 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.067 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:22:22.068 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:22:22.069 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:22:22.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:22:22.071 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.073 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.074 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:22:22.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.076 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.077 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:22:22.077 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:22:22.078 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 10 librados: watch_flush enter 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 10 librados: watch_flush exit 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 20 client.84422.objecter shutdown clearing up homeless session... 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 10 client.84422.objecter successfully canceled tick 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 >> v2:172.21.15.190:6800/1562782189 conn(0x7f486003cef0 msgr2=0x7f486003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 --2- 172.21.15.138:0/84783805 >> v2:172.21.15.190:6800/1562782189 conn(0x7f486003cef0 0x7f486003f3b0 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 crypto rx=0x7f4868009740 tx=0x7f48680a6000 comp rx=0 tx=0).stop 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 10 monclient: shutdown 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488013d640 msgr2=0x7f48801367a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 --2- 172.21.15.138:0/84783805 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f488013d640 0x7f48801367a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f487000c3e0 tx=0x7f487000c8a0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f487ffff640 1 -- 172.21.15.138:0/84783805 reap_dead start 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f487ffff640 1 -- 172.21.15.138:0/84783805 reap_dead start 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 shutdown_connections 2024-09-11T05:22:22.079 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 >> 172.21.15.138:0/84783805 conn(0x7f48800ab830 msgr2=0x7f488006ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:22.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 shutdown_connections 2024-09-11T05:22:22.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 -- 172.21.15.138:0/84783805 wait complete. 2024-09-11T05:22:22.080 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.060+0000 7f4885df2640 1 librados: shutdown 2024-09-11T05:22:22.080 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:22:22.080 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:22:22.080 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:22:22.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient: get_monmap_and_config 2024-09-11T05:22:22.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient: build_initial_monmap 2024-09-11T05:22:22.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient: monmap: 2024-09-11T05:22:22.192 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:22.192 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.192 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:22.198105+0000 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:22.198105+0000 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:22.193 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 1 Processor -- start 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 1 -- start start 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient: init 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c070010 0x7f120c135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient(hunting): picked mon.noname-b con 0x7f120c070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f120c09e6e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f120c09eae0 0x7f120c1389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient(hunting): picked mon.noname-a con 0x7f120c09eae0 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f120c0a8990 con 0x7f120c09e6e0 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f120c0703f0 con 0x7f120c09eae0 2024-09-11T05:22:22.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f120c09cbf0 con 0x7f120c070010 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121226f640 10 monclient(hunting): authenticate will time out at 1825.634978s 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c070010 0x7f120c135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f120c09eae0 0x7f120c1389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c070010 0x7f120c135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48486/0 (socket says 172.21.15.138:48486) 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 1 -- 172.21.15.138:0/2331305004 learned_addr learned my addr 172.21.15.138:0/2331305004 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 10 monclient(hunting): get_auth_request con 0x7f120c070010 auth_method 0 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient(hunting): get_auth_request con 0x7f120c09eae0 auth_method 0 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient(hunting): handle_auth_done global_id 114114 payload 274 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 1 -- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c070010 msgr2=0x7f120c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 1 --2- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c070010 0x7f120c135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 1 -- 172.21.15.138:0/2331305004 >> v1:172.21.15.190:6789/0 conn(0x7f120c09e6e0 legacy=0x7f120c1382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient: _finish_hunting 0 2024-09-11T05:22:22.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 1 monclient: found mon.noname-a 2024-09-11T05:22:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f121080c640 1 -- 172.21.15.138:0/2331305004 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120c09ced0 con 0x7f120c09eae0 2024-09-11T05:22:22.199 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.196+0000 7f120bfff640 1 --2- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c070010 0x7f120c135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 1 -- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c09eae0 msgr2=0x7f120c1389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.200 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 1 --2- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c09eae0 0x7f120c1389d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1204009d90 tx=0x7f120401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a62dc76b76602b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 1 -- 172.21.15.138:0/2331305004 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12040b2070 con 0x7f120c09eae0 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: dump: 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.201 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: _finish_auth 0 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 1 -- 172.21.15.138:0/2331305004 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12040a58f0 con 0x7f120c09eae0 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 1 -- 172.21.15.138:0/2331305004 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12040a5cc0 con 0x7f120c09eae0 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 5 monclient: authenticate success, global_id 114114 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 4 set_mon_vals no callback set 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120affd640 10 monclient: dump: 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.202 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient: shutdown 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c09eae0 msgr2=0x7f120c1389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 --2- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c09eae0 0x7f120c1389d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1204009d90 tx=0x7f120401c890 comp rx=0 tx=0).stop 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- 172.21.15.138:0/2331305004 shutdown_connections 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 --2- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c09eae0 0x7f120c1389d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.203 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 --2- 172.21.15.138:0/2331305004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c070010 0x7f120c135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- 172.21.15.138:0/2331305004 >> 172.21.15.138:0/2331305004 conn(0x7f120c0ab830 msgr2=0x7f120c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- 172.21.15.138:0/2331305004 shutdown_connections 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- 172.21.15.138:0/2331305004 wait complete. 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient: build_initial_monmap 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient: monmap: 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:22.204 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:22.205 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 librados: starting msgr at 2024-09-11T05:22:22.209 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 librados: starting objecter 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 Processor -- start 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- start start 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 librados: setting wanted keys 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 librados: calling monclient init 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient: init 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c070010 0x7f120c131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient(hunting): picked mon.noname-a con 0x7f120c070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f120c09e6e0 0x7f120c0a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient(hunting): picked mon.noname-b con 0x7f120c09e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c09eae0 0x7f120c0a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.210 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient(hunting): picked mon.noname-c con 0x7f120c09eae0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f120c142e80 con 0x7f120c070010 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f120c139120 con 0x7f120c09e6e0 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f120c143000 con 0x7f120c09eae0 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121226f640 10 monclient(hunting): authenticate will time out at 1825.642504s 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c070010 0x7f120c131e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c070010 0x7f120c131e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45592/0 (socket says 172.21.15.138:45592) 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 -- 172.21.15.138:0/2514923650 learned_addr learned my addr 172.21.15.138:0/2514923650 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient(hunting): get_auth_request con 0x7f120c070010 auth_method 0 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.211 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c09eae0 0x7f120c0a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 10 monclient(hunting): get_auth_request con 0x7f120c09eae0 auth_method 0 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f121080c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient(hunting): handle_auth_done global_id 114117 payload 824 2024-09-11T05:22:22.212 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c09eae0 msgr2=0x7f120c0a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c09eae0 0x7f120c0a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f120c09e6e0 msgr2=0x7f120c0a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f120c09e6e0 0x7f120c0a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 monclient: found mon.noname-a 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.204+0000 7f120bfff640 1 -- 172.21.15.138:0/2514923650 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120c0a2330 con 0x7f120c070010 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120bfff640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c070010 msgr2=0x7f120c131e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120bfff640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c070010 0x7f120c131e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00e9d0 tx=0x7f11fc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 client.?.objecter ms_handle_connect 0x7f120c070010 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:22.213 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc017070 con 0x7f120c070010 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: dump: 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: _finish_auth 0 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11fc00cc70 con 0x7f120c070010 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 5 monclient: authenticate success, global_id 114117 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 10 monclient: _renew_subs 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120c1334f0 con 0x7f120c070010 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc013670 con 0x7f120c070010 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 monclient: dump: 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.215 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 10 client.114117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 10 monclient: _renew_subs 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f120c139a70 con 0x7f120c070010 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 1 librados: init done 2024-09-11T05:22:22.216 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1209ffb640 4 set_mon_vals no callback set 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 0x7f120c0a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f120c0aaae0 con 0x7f120c132380 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 0x7f120c0a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 10 monclient: get_auth_request con 0x7f120c132380 auth_method 0 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f11fc024060 con 0x7f120c070010 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 10 monclient(hunting): handle_auth_done global_id 84431 payload 824 2024-09-11T05:22:22.217 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 1 -- 172.21.15.138:0/2514923650 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f120c135450 con 0x7f120c132380 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 msgr2=0x7f120c0a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 msgr2=0x7f120c0a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 0x7f120c0a0fe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f120c06efb0 tx=0x7f120000c7a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f120b7fe640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 0x7f120c0a0fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 1 --2- 172.21.15.138:0/2514923650 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11f803cef0 0x7f11f803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(611..611 src has 520..611) ==== 5977+0+0 (secure 0 0 0) 0x7f11fc04fb90 con 0x7f120c070010 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 10 client.114117.objecter ms_dispatch 0x7f120c000ba0 osd_map(611..611 src has 520..611) 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 3 client.114117.objecter handle_osd_map got epochs [611,611] > 0 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 3 client.114117.objecter handle_osd_map decoding full epoch 611 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f1208ff9640 20 client.114117.objecter dump_active .. 0 homeless 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121080c640 1 --2- 172.21.15.138:0/2514923650 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11f803cef0 0x7f11f803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.208+0000 7f121080c640 10 monclient: get_auth_request con 0x7f11f803cef0 auth_method 0 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.212+0000 7f1208ff9640 10 client.114117.objecter ms_handle_connect 0x7f11f803cef0 2024-09-11T05:22:22.218 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.212+0000 7f121080c640 1 --2- 172.21.15.138:0/2514923650 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11f803cef0 0x7f11f803f3b0 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 crypto rx=0x7f120c0705b0 tx=0x7f12040a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 0x7f120c0a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 10 monclient: get_auth_request con 0x7f120c132380 auth_method 0 2024-09-11T05:22:22.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.407 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 10 monclient: _init_auth method 2 2024-09-11T05:22:22.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 10 monclient: _init_auth already have auth, reseting 2024-09-11T05:22:22.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 10 monclient: handle_auth_reply_more payload 9 2024-09-11T05:22:22.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-11T05:22:22.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 10 monclient: handle_auth_done global_id 84431 payload 293 2024-09-11T05:22:22.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 msgr2=0x7f120c0a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f120b7fe640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 0x7f120c0a0fe0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1200005630 tx=0x7f1200007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 10 client.114117.objecter ms_handle_connect 0x7f120c132380 2024-09-11T05:22:22.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 10 client.114117.objecter resend_mon_ops 2024-09-11T05:22:22.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200018030 con 0x7f120c132380 2024-09-11T05:22:22.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:22.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 10 monclient: dump: 2024-09-11T05:22:22.411 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.411 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f120000fc00 con 0x7f120c132380 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 msgr2=0x7f120c0a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 0x7f120c0a0fe0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1200005630 tx=0x7f1200007dc0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.412 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 msgr2=0x7f120c0a0fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:22.413 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.412+0000 7f1208ff9640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c132380 0x7f120c0a0fe0 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1200005630 tx=0x7f1200007dc0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f121226f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:22:22.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f121226f640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c0a1520 0x7f120c0a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f121226f640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f120c13bc50 con 0x7f120c0a1520 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c0a1520 0x7f120c0a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 10 monclient: get_auth_request con 0x7f120c0a1520 auth_method 0 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 10 monclient(hunting): handle_auth_done global_id 84434 payload 824 2024-09-11T05:22:22.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.420+0000 7f120b7fe640 1 -- 172.21.15.138:0/2514923650 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f120c069fa0 con 0x7f120c0a1520 2024-09-11T05:22:22.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f120b7fe640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c0a1520 msgr2=0x7f120c0a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f120b7fe640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c0a1520 0x7f120c0a2540 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f120c13a8c0 tx=0x7f12000037e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 10 client.114117.objecter ms_handle_connect 0x7f120c0a1520 2024-09-11T05:22:22.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 10 client.114117.objecter resend_mon_ops 2024-09-11T05:22:22.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120000f4b0 con 0x7f120c0a1520 2024-09-11T05:22:22.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:22.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 10 monclient: dump: 2024-09-11T05:22:22.422 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.422 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.422 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f120000f410 con 0x7f120c0a1520 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c0a1520 msgr2=0x7f120c0a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c0a1520 0x7f120c0a2540 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f120c13a8c0 tx=0x7f12000037e0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 58, 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 386194, 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.424 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.425 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.426 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:22:22.427 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.428 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.429 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.430 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.431 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.432 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.433 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.434 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.435 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.436 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:22:22.436 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.436 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.436 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:22:22.436 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:22:22.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c0a1520 msgr2=0x7f120c0a2540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:22.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f1208ff9640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f120c0a1520 0x7f120c0a2540 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 10 librados: watch_flush enter 2024-09-11T05:22:22.437 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 10 librados: watch_flush exit 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 20 client.114117.objecter shutdown clearing up homeless session... 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 10 client.114117.objecter successfully canceled tick 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11f803cef0 msgr2=0x7f11f803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 --2- 172.21.15.138:0/2514923650 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11f803cef0 0x7f11f803f3b0 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 crypto rx=0x7f120c0705b0 tx=0x7f12040a7070 comp rx=0 tx=0).stop 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 10 monclient: shutdown 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c070010 msgr2=0x7f120c131e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 --2- 172.21.15.138:0/2514923650 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f120c070010 0x7f120c131e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11fc00e9d0 tx=0x7f11fc00ee90 comp rx=0 tx=0).stop 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121080c640 1 -- 172.21.15.138:0/2514923650 reap_dead start 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121080c640 1 -- 172.21.15.138:0/2514923650 reap_dead start 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 shutdown_connections 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 >> 172.21.15.138:0/2514923650 conn(0x7f120c0ab830 msgr2=0x7f120c0a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 shutdown_connections 2024-09-11T05:22:22.438 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 -- 172.21.15.138:0/2514923650 wait complete. 2024-09-11T05:22:22.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.424+0000 7f121226f640 1 librados: shutdown 2024-09-11T05:22:22.447 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:22:22.447 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:22:22.447 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:22:22.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient: get_monmap_and_config 2024-09-11T05:22:22.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient: build_initial_monmap 2024-09-11T05:22:22.572 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient: monmap: 2024-09-11T05:22:22.572 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:22.578191+0000 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:22.578191+0000 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:22.573 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 1 Processor -- start 2024-09-11T05:22:22.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 1 -- start start 2024-09-11T05:22:22.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient: init 2024-09-11T05:22:22.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:22.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-11T05:22:22.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f26340a32c0 0x7f263409e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.575 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient(hunting): picked mon.noname-a con 0x7f26340a32c0 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f263409e960 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f263409ed80 0x7f2634135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient(hunting): picked mon.noname-c con 0x7f263409ed80 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26340a1b50 con 0x7f263409e960 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f26340a1920 con 0x7f26340a32c0 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f26340a36a0 con 0x7f263409ed80 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2638f00640 10 monclient(hunting): authenticate will time out at 1826.014259s 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f26340a32c0 0x7f263409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2631d9b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f263409e960 0x7f26341356f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45636/0 (socket says 172.21.15.138:45636) 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient(hunting): get_auth_request con 0x7f26340a32c0 auth_method 0 2024-09-11T05:22:22.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f2631d9b640 1 -- 172.21.15.138:0/1665469455 learned_addr learned my addr 172.21.15.138:0/1665469455 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient(hunting): handle_auth_done global_id 114120 payload 274 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 1 -- 172.21.15.138:0/1665469455 >> v2:172.21.15.190:3300/0 conn(0x7f263409ed80 msgr2=0x7f2634135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 1 --2- 172.21.15.138:0/1665469455 >> v2:172.21.15.190:3300/0 conn(0x7f263409ed80 0x7f2634135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 1 -- 172.21.15.138:0/1665469455 >> v1:172.21.15.138:6789/0 conn(0x7f263409e960 legacy=0x7f26341356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient: _finish_hunting 0 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 1 monclient: found mon.noname-a 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.576+0000 7f263259c640 1 -- 172.21.15.138:0/1665469455 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263409f560 con 0x7f26340a32c0 2024-09-11T05:22:22.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263259c640 1 -- 172.21.15.138:0/1665469455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26340a32c0 msgr2=0x7f263409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263259c640 1 --2- 172.21.15.138:0/1665469455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26340a32c0 0x7f263409e420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f26340a2000 tx=0x7f262401e520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548919b817cbf47f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 1 -- 172.21.15.138:0/1665469455 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26240b3070 con 0x7f26340a32c0 2024-09-11T05:22:22.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:22.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: dump: 2024-09-11T05:22:22.580 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.580 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: _finish_auth 0 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: _check_auth_tickets 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 5 monclient: authenticate success, global_id 114120 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:22.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 1 -- 172.21.15.138:0/1665469455 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26240ae620 con 0x7f26340a32c0 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 1 -- 172.21.15.138:0/1665469455 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26240ae9f0 con 0x7f26340a32c0 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 4 set_mon_vals no callback set 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f263159a640 10 monclient: dump: 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.582 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 10 monclient: shutdown 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 1 -- 172.21.15.138:0/1665469455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26340a32c0 msgr2=0x7f263409e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 1 --2- 172.21.15.138:0/1665469455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26340a32c0 0x7f263409e420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f26340a2000 tx=0x7f262401e520 comp rx=0 tx=0).stop 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 1 -- 172.21.15.138:0/1665469455 shutdown_connections 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 1 --2- 172.21.15.138:0/1665469455 >> v2:172.21.15.190:3300/0 conn(0x7f263409ed80 0x7f2634135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 1 --2- 172.21.15.138:0/1665469455 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f26340a32c0 0x7f263409e420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 1 -- 172.21.15.138:0/1665469455 >> 172.21.15.138:0/1665469455 conn(0x7f2634063640 msgr2=0x7f2634063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 1 -- 172.21.15.138:0/1665469455 shutdown_connections 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.580+0000 7f2638f00640 1 -- 172.21.15.138:0/1665469455 wait complete. 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient: build_initial_monmap 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient: monmap: 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:22.584 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 librados: starting msgr at 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 librados: starting objecter 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 Processor -- start 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 -- start start 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 librados: setting wanted keys 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 librados: calling monclient init 2024-09-11T05:22:22.585 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient: init 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f263409e960 0x7f263413ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient(hunting): picked mon.noname-a con 0x7f263409e960 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f263409ed80 0x7f263413b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient(hunting): picked mon.noname-c con 0x7f263409ed80 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340a32c0 0x7f263413b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient(hunting): picked mon.noname-b con 0x7f26340a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:22.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f26340a12d0 con 0x7f263409e960 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f263409f190 con 0x7f26340a32c0 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2634073f00 con 0x7f263409ed80 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2638f00640 10 monclient(hunting): authenticate will time out at 1826.021467s 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f263409e960 0x7f263413ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f263409ed80 0x7f263413b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f263409ed80 0x7f263413b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48520/0 (socket says 172.21.15.138:48520) 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 -- 172.21.15.138:0/3521982165 learned_addr learned my addr 172.21.15.138:0/3521982165 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f263409e960 0x7f263413ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45600/0 (socket says 172.21.15.138:45600) 2024-09-11T05:22:22.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient(hunting): get_auth_request con 0x7f263409ed80 auth_method 0 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 10 monclient(hunting): get_auth_request con 0x7f263409e960 auth_method 0 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient(hunting): handle_auth_done global_id 84437 payload 824 2024-09-11T05:22:22.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340a32c0 msgr2=0x7f263413b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2632d9d640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340a32c0 0x7f263413b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340a32c0 0x7f263413b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f263409e960 msgr2=0x7f263413ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f263409e960 0x7f263413ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 monclient: found mon.noname-c 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2631d9b640 1 -- 172.21.15.138:0/3521982165 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263409e110 con 0x7f263409ed80 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2632d9d640 10 monclient: get_auth_request con 0x7f26340a32c0 auth_method 0 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f263259c640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f263409e960 0x7f263413ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.584+0000 7f2632d9d640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26340a32c0 0x7f263413b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:22:22.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2631d9b640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f263409ed80 msgr2=0x7f263413b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2631d9b640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f263409ed80 0x7f263413b0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f262800b430 tx=0x7f262800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 client.?.objecter ms_handle_connect 0x7f263409ed80 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628015020 con 0x7f263409ed80 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: dump: 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.595 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: _finish_auth 0 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2628011a60 con 0x7f263409ed80 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628011e50 con 0x7f263409ed80 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 5 monclient: authenticate success, global_id 84437 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 10 monclient: _renew_subs 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26341235d0 con 0x7f263409ed80 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2630d99640 4 set_mon_vals no callback set 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 monclient: dump: 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 10 client.84437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 10 monclient: _renew_subs 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:22.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2634122a40 con 0x7f263409ed80 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 1 librados: init done 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d0ef0 0x7f263402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f26340a1450 con 0x7f26341d0ef0 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f262801a400 con 0x7f263409ed80 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 1 --2- 172.21.15.138:0/3521982165 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261403d070 0x7f261403f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(611..611 src has 520..611) ==== 5977+0+0 (secure 0 0 0) 0x7f262804fdf0 con 0x7f263409ed80 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 10 client.84437.objecter ms_dispatch 0x7f2634000ba0 osd_map(611..611 src has 520..611) 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 3 client.84437.objecter handle_osd_map got epochs [611,611] > 0 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 3 client.84437.objecter handle_osd_map decoding full epoch 611 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d0ef0 0x7f263402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 10 monclient: get_auth_request con 0x7f26341d0ef0 auth_method 0 2024-09-11T05:22:22.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2622ffd640 20 client.84437.objecter dump_active .. 0 homeless 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2632d9d640 1 --2- 172.21.15.138:0/3521982165 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261403d070 0x7f261403f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f2632d9d640 10 monclient: get_auth_request con 0x7f261403d070 auth_method 0 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 10 monclient(hunting): handle_auth_done global_id 104170 payload 824 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.588+0000 7f263259c640 1 -- 172.21.15.138:0/3521982165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f263409e420 con 0x7f26341d0ef0 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f263259c640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d0ef0 msgr2=0x7f263402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f263259c640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d0ef0 0x7f263402d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f26240b0000 tx=0x7f262401b2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 10 client.84437.objecter ms_handle_connect 0x7f26341d0ef0 2024-09-11T05:22:22.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 10 client.84437.objecter resend_mon_ops 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 10 client.84437.objecter ms_handle_connect 0x7f261403d070 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2632d9d640 1 --2- 172.21.15.138:0/3521982165 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261403d070 0x7f261403f530 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7f26341219c0 tx=0x7f262c007340 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26240bd070 con 0x7f26341d0ef0 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 10 monclient: dump: 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.600 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26240b8660 con 0x7f26341d0ef0 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d0ef0 msgr2=0x7f263402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d0ef0 0x7f263402d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f26240b0000 tx=0x7f262401b2c0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d0ef0 msgr2=0x7f263402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:22.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.592+0000 7f2622ffd640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d0ef0 0x7f263402d450 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f26240b0000 tx=0x7f262401b2c0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.602 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.596+0000 7f2638f00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.596+0000 7f2638f00640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d12d0 0x7f263402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.596+0000 7f2638f00640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.596+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f26341233b0 con 0x7f26341d12d0 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d12d0 0x7f263402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 10 monclient: get_auth_request con 0x7f26341d12d0 auth_method 0 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 10 monclient(hunting): handle_auth_done global_id 104173 payload 824 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 1 -- 172.21.15.138:0/3521982165 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2634138020 con 0x7f26341d12d0 2024-09-11T05:22:22.603 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d12d0 msgr2=0x7f263402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f2622ffd640 10 client.84437.objecter ms_handle_connect 0x7f26341d12d0 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f2622ffd640 10 client.84437.objecter resend_mon_ops 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f263259c640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d12d0 0x7f263402d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f262401cbd0 tx=0x7f2624009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26240b3040 con 0x7f26341d12d0 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f2622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f2622ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.600+0000 7f2622ffd640 10 monclient: dump: 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.604 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f2624003a30 con 0x7f26341d12d0 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2622ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2622ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d12d0 msgr2=0x7f263402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2622ffd640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d12d0 0x7f263402d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f262401cbd0 tx=0x7f2624009190 comp rx=0 tx=0).stop 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2622ffd640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d12d0 msgr2=0x7f263402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2622ffd640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26341d12d0 0x7f263402d990 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f262401cbd0 tx=0x7f2624009190 comp rx=0 tx=0).stop 2024-09-11T05:22:22.605 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 58, 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 132763, 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:22:22.606 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:22:22.607 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.608 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:22:22.609 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:22.610 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.613 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:22.614 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:22:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:22:22.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:22.617 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:22.618 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:22.618 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:22.618 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:22.618 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:22.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:22.618 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:22:22.618 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:22:22.619 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 10 librados: watch_flush enter 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 10 librados: watch_flush exit 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 20 client.84437.objecter shutdown clearing up homeless session... 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 10 client.84437.objecter successfully canceled tick 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261403d070 msgr2=0x7f261403f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 --2- 172.21.15.138:0/3521982165 >> v2:172.21.15.190:6800/1562782189 conn(0x7f261403d070 0x7f261403f530 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7f26341219c0 tx=0x7f262c007340 comp rx=0 tx=0).stop 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 10 monclient: shutdown 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f263409ed80 msgr2=0x7f263413b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f263409ed80 0x7f263413b0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f262800b430 tx=0x7f262800b8f0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2632d9d640 1 -- 172.21.15.138:0/3521982165 reap_dead start 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 shutdown_connections 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 --2- 172.21.15.138:0/3521982165 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f263409ed80 0x7f263413b0c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 >> 172.21.15.138:0/3521982165 conn(0x7f2634063640 msgr2=0x7f263413cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:22.620 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 shutdown_connections 2024-09-11T05:22:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 -- 172.21.15.138:0/3521982165 wait complete. 2024-09-11T05:22:22.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.604+0000 7f2638f00640 1 librados: shutdown 2024-09-11T05:22:22.621 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:22:22.621 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:22:22.621 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:22:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.732+0000 7fe99a19a640 10 monclient: get_monmap_and_config 2024-09-11T05:22:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.732+0000 7fe99a19a640 10 monclient: build_initial_monmap 2024-09-11T05:22:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.732+0000 7fe99a19a640 10 monclient: monmap: 2024-09-11T05:22:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.730 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:22.736080+0000 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:22.736080+0000 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:22.731 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.732+0000 7fe99a19a640 1 Processor -- start 2024-09-11T05:22:22.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.732+0000 7fe99a19a640 1 -- start start 2024-09-11T05:22:22.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.732+0000 7fe99a19a640 10 monclient: init 2024-09-11T05:22:22.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.732+0000 7fe99a19a640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:22.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-11T05:22:22.733 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe99406e0e0 0x7fe994136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 10 monclient(hunting): picked mon.noname-b con 0x7fe99406e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe994136170 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe994136c60 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9940a3270 con 0x7fe994136c60 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9940a0af0 con 0x7fe994136170 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe99406e4c0 con 0x7fe99406e0e0 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe99a19a640 10 monclient(hunting): authenticate will time out at 1826.172113s 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe99406e0e0 0x7fe994136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe993fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fe994136c60 0x7fe9940a2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45646/0 (socket says 172.21.15.138:45646) 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe993fff640 1 -- 172.21.15.138:0/1482327810 learned_addr learned my addr 172.21.15.138:0/1482327810 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:22:22.734 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient(hunting): get_auth_request con 0x7fe99406e0e0 auth_method 0 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient(hunting): handle_auth_done global_id 84440 payload 274 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 1 -- 172.21.15.138:0/1482327810 >> v1:172.21.15.190:6789/0 conn(0x7fe994136170 legacy=0x7fe9940a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 1 -- 172.21.15.138:0/1482327810 >> v1:172.21.15.138:6789/0 conn(0x7fe994136c60 legacy=0x7fe9940a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient: _finish_hunting 0 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 1 monclient: found mon.noname-b 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:22.735 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:22.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.736+0000 7fe9937fe640 1 -- 172.21.15.138:0/1482327810 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9940a2740 con 0x7fe99406e0e0 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 -- 172.21.15.138:0/1482327810 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe99406e0e0 msgr2=0x7fe994136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 --2- 172.21.15.138:0/1482327810 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe99406e0e0 0x7fe994136720 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe9880097f0 tx=0x7fe98801c1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5ae7fc94feee8b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 1 -- 172.21.15.138:0/1482327810 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880b0070 con 0x7fe99406e0e0 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: dump: 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.740 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: _finish_auth 0 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: _check_auth_tickets 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 1 -- 172.21.15.138:0/1482327810 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9880ab600 con 0x7fe99406e0e0 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 5 monclient: authenticate success, global_id 84440 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:22.741 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 4 set_mon_vals no callback set 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 1 -- 172.21.15.138:0/1482327810 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880ab9d0 con 0x7fe99406e0e0 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9927fc640 10 monclient: dump: 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.742 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.743 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.743 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.743 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient: shutdown 2024-09-11T05:22:22.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:22.747 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- 172.21.15.138:0/1482327810 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe99406e0e0 msgr2=0x7fe994136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 --2- 172.21.15.138:0/1482327810 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe99406e0e0 0x7fe994136720 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe9880097f0 tx=0x7fe98801c1c0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- 172.21.15.138:0/1482327810 shutdown_connections 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 --2- 172.21.15.138:0/1482327810 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe99406e0e0 0x7fe994136720 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- 172.21.15.138:0/1482327810 >> 172.21.15.138:0/1482327810 conn(0x7fe994063640 msgr2=0x7fe994063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- 172.21.15.138:0/1482327810 shutdown_connections 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- 172.21.15.138:0/1482327810 wait complete. 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient: build_initial_monmap 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient: monmap: 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.748 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 librados: starting msgr at 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 librados: starting objecter 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 Processor -- start 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- start start 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 librados: setting wanted keys 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 librados: calling monclient init 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient: init 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe99406e0e0 0x7fe9941376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient(hunting): picked mon.noname-a con 0x7fe99406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe994136170 0x7fe994137c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient(hunting): picked mon.noname-c con 0x7fe994136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe994136c60 0x7fe9941d0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient(hunting): picked mon.noname-b con 0x7fe994136c60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe9941420b0 con 0x7fe99406e0e0 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe994141f30 con 0x7fe994136c60 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe994142230 con 0x7fe994136170 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe99a19a640 10 monclient(hunting): authenticate will time out at 1826.183182s 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe99406e0e0 0x7fe9941376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe994136170 0x7fe994137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe99406e0e0 0x7fe9941376f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45614/0 (socket says 172.21.15.138:45614) 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe994136170 0x7fe994137c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48534/0 (socket says 172.21.15.138:48534) 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 -- 172.21.15.138:0/4118132280 learned_addr learned my addr 172.21.15.138:0/4118132280 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient(hunting): get_auth_request con 0x7fe99406e0e0 auth_method 0 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 10 monclient(hunting): get_auth_request con 0x7fe994136170 auth_method 0 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe992ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe993fff640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe994136c60 0x7fe9941d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient(hunting): handle_auth_done global_id 114126 payload 824 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 -- 172.21.15.138:0/4118132280 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe994136170 msgr2=0x7fe994137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe994136170 0x7fe994137c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 -- 172.21.15.138:0/4118132280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe994136c60 msgr2=0x7fe9941d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe994136c60 0x7fe9941d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient: _finish_hunting 0 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 monclient: found mon.noname-a 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe9937fe640 1 -- 172.21.15.138:0/4118132280 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe994139780 con 0x7fe99406e0e0 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe993fff640 10 monclient: get_auth_request con 0x7fe994136c60 auth_method 0 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.744+0000 7fe993fff640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe994136c60 0x7fe9941d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe9937fe640 1 -- 172.21.15.138:0/4118132280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe99406e0e0 msgr2=0x7fe9941376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe9937fe640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe99406e0e0 0x7fe9941376f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe98801ca60 tx=0x7fe988018f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 client.?.objecter ms_handle_connect 0x7fe99406e0e0 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 1 -- 172.21.15.138:0/4118132280 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880b9070 con 0x7fe99406e0e0 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: dump: 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: _finish_auth 0 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 1 -- 172.21.15.138:0/4118132280 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9880b5690 con 0x7fe99406e0e0 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 5 monclient: authenticate success, global_id 114126 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 10 monclient: _renew_subs 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe994134270 con 0x7fe99406e0e0 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 1 -- 172.21.15.138:0/4118132280 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880b0040 con 0x7fe99406e0e0 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 monclient: dump: 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe991ffb640 4 set_mon_vals no callback set 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 10 client.114126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 10 monclient: _renew_subs 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe99409e110 con 0x7fe99406e0e0 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 1 librados: init done 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 1 -- 172.21.15.138:0/4118132280 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fe9880b5aa0 con 0x7fe99406e0e0 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe994136720 con 0x7fe99406e0e0 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 1 --2- 172.21.15.138:0/4118132280 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe98003cef0 0x7fe98003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 1 -- 172.21.15.138:0/4118132280 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(611..611 src has 520..611) ==== 5977+0+0 (secure 0 0 0) 0x7fe9880ea310 con 0x7fe99406e0e0 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 10 client.114126.objecter ms_dispatch 0x7fe994000ba0 osd_map(611..611 src has 520..611) 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 3 client.114126.objecter handle_osd_map got epochs [611,611] > 0 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 3 client.114126.objecter handle_osd_map decoding full epoch 611 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe97bfff640 20 client.114126.objecter dump_active .. 0 homeless 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.748+0000 7fe992ffd640 1 --2- 172.21.15.138:0/4118132280 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe98003cef0 0x7fe98003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.752+0000 7fe992ffd640 10 monclient: get_auth_request con 0x7fe98003cef0 auth_method 0 2024-09-11T05:22:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.752+0000 7fe97bfff640 10 client.114126.objecter ms_handle_connect 0x7fe98003cef0 2024-09-11T05:22:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.752+0000 7fe992ffd640 1 --2- 172.21.15.138:0/4118132280 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe98003cef0 0x7fe98003f3b0 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7fe984007fd0 tx=0x7fe984007e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.752+0000 7fe97bfff640 1 -- 172.21.15.138:0/4118132280 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe9940a3270 con 0x7fe99406e0e0 2024-09-11T05:22:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.752+0000 7fe97bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:22.752+0000 7fe97bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:23.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe99a19a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:22:23.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe99a19a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:22:23.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe99a19a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:23.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe994139d60 con 0x7fe99406e0e0 2024-09-11T05:22:23.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe97bfff640 1 -- 172.21.15.138:0/4118132280 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe9940a0af0 con 0x7fe99406e0e0 2024-09-11T05:22:23.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe97bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:22:23.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe97bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:23.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=2425395877,config=692619927,health=3108313556,kv=3356816980,logm=4142670995}) 2024-09-11T05:22:23.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=10,kv=10,logm=59} crc {auth=3194886319,config=692619927,health=167970060,kv=310889196,logm=3618839452}) 2024-09-11T05:22:23.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=2425395877,config=692619927,health=3108313556,kv=3356816980,logm=4142670995}) 2024-09-11T05:22:23.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=45941663,config=692619927,health=2540375241,kv=310889196,logm=402992327}) 2024-09-11T05:22:23.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 10 librados: watch_flush enter 2024-09-11T05:22:23.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 10 librados: watch_flush exit 2024-09-11T05:22:23.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 20 client.114126.objecter shutdown clearing up homeless session... 2024-09-11T05:22:23.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 10 client.114126.objecter successfully canceled tick 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe98003cef0 msgr2=0x7fe98003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 --2- 172.21.15.138:0/4118132280 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe98003cef0 0x7fe98003f3b0 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7fe984007fd0 tx=0x7fe984007e40 comp rx=0 tx=0).stop 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 10 monclient: shutdown 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe99406e0e0 msgr2=0x7fe9941376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe99406e0e0 0x7fe9941376f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe98801ca60 tx=0x7fe988018f60 comp rx=0 tx=0).stop 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 shutdown_connections 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 --2- 172.21.15.138:0/4118132280 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe98003cef0 0x7fe98003f3b0 unknown :-1 s=CLOSED pgs=3480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe994136c60 0x7fe9941d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe994136170 0x7fe994137c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 --2- 172.21.15.138:0/4118132280 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe99406e0e0 0x7fe9941376f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 >> 172.21.15.138:0/4118132280 conn(0x7fe994063640 msgr2=0x7fe994135290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:23.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 shutdown_connections 2024-09-11T05:22:23.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 -- 172.21.15.138:0/4118132280 wait complete. 2024-09-11T05:22:23.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe99a19a640 1 librados: shutdown 2024-09-11T05:22:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750427415}) 2024-09-11T05:22:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2971242660}) 2024-09-11T05:22:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750427415}) 2024-09-11T05:22:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1863633080}) 2024-09-11T05:22:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733817337}) 2024-09-11T05:22:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1113715968}) 2024-09-11T05:22:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733817337}) 2024-09-11T05:22:23.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961944028}) 2024-09-11T05:22:23.034 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:22:23.431 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.432+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.431 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.432+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938189335}) 2024-09-11T05:22:23.431 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.432+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3561200372}) 2024-09-11T05:22:23.431 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.432+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.431 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.432+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938189335}) 2024-09-11T05:22:23.431 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.432+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519445271}) 2024-09-11T05:22:23.627 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.627 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50951252}) 2024-09-11T05:22:23.628 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159242658}) 2024-09-11T05:22:23.628 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.628 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50951252}) 2024-09-11T05:22:23.628 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1919166595}) 2024-09-11T05:22:23.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.952+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.952+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986422479}) 2024-09-11T05:22:23.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.952+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469951929}) 2024-09-11T05:22:23.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.952+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.950 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.952+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986422479}) 2024-09-11T05:22:23.950 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.952+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1773642245}) 2024-09-11T05:22:23.956 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.960+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.956 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.960+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780246560}) 2024-09-11T05:22:23.956 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.960+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2287258441}) 2024-09-11T05:22:23.956 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.960+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.956 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.960+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780246560}) 2024-09-11T05:22:23.956 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.960+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2960505606}) 2024-09-11T05:22:23.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.972+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.972+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1231239712}) 2024-09-11T05:22:23.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.972+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878608848}) 2024-09-11T05:22:23.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.972+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.972+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1231239712}) 2024-09-11T05:22:23.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.972+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159666976}) 2024-09-11T05:22:23.975 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.976+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.975 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.976+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747388656}) 2024-09-11T05:22:23.975 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.976+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=88452858}) 2024-09-11T05:22:23.975 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.976 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747388656}) 2024-09-11T05:22:23.976 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1294365622}) 2024-09-11T05:22:23.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963110159}) 2024-09-11T05:22:23.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376400885}) 2024-09-11T05:22:23.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963110159}) 2024-09-11T05:22:23.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.980+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618375016}) 2024-09-11T05:22:23.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.984+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.984+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690117421}) 2024-09-11T05:22:23.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.984+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=873739124}) 2024-09-11T05:22:23.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.984+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.984+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690117421}) 2024-09-11T05:22:23.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.984+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1303628230}) 2024-09-11T05:22:23.984 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.984 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351817058}) 2024-09-11T05:22:23.984 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4198482269}) 2024-09-11T05:22:23.984 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.984 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351817058}) 2024-09-11T05:22:23.984 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2820446512}) 2024-09-11T05:22:23.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2164303706}) 2024-09-11T05:22:23.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413817650}) 2024-09-11T05:22:23.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2164303706}) 2024-09-11T05:22:23.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.988+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3819425728}) 2024-09-11T05:22:23.990 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.992+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.990 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.992+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331533909}) 2024-09-11T05:22:23.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.992+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249706889}) 2024-09-11T05:22:23.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.992+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.992+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331533909}) 2024-09-11T05:22:23.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.992+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2326928314}) 2024-09-11T05:22:23.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.996+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.996+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046014069}) 2024-09-11T05:22:23.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.996+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3938345376}) 2024-09-11T05:22:23.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.996+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.996+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046014069}) 2024-09-11T05:22:23.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:23.996+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1068356557}) 2024-09-11T05:22:23.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3816685658}) 2024-09-11T05:22:23.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2796958994}) 2024-09-11T05:22:23.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3816685658}) 2024-09-11T05:22:23.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4051190168}) 2024-09-11T05:22:23.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605714351}) 2024-09-11T05:22:23.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=715403203}) 2024-09-11T05:22:23.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:23.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605714351}) 2024-09-11T05:22:23.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.000+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=222044787}) 2024-09-11T05:22:24.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.004+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.004+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3164686002}) 2024-09-11T05:22:24.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.004+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685784411}) 2024-09-11T05:22:24.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.004+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.004+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3164686002}) 2024-09-11T05:22:24.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.004+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772590180}) 2024-09-11T05:22:24.004 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.004 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687770425}) 2024-09-11T05:22:24.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3829538437}) 2024-09-11T05:22:24.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687770425}) 2024-09-11T05:22:24.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587600694}) 2024-09-11T05:22:24.006 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092065459}) 2024-09-11T05:22:24.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045135995}) 2024-09-11T05:22:24.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092065459}) 2024-09-11T05:22:24.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.008+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347614674}) 2024-09-11T05:22:24.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=279070284}) 2024-09-11T05:22:24.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1067251916}) 2024-09-11T05:22:24.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=279070284}) 2024-09-11T05:22:24.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=919829111}) 2024-09-11T05:22:24.011 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.011 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788489928}) 2024-09-11T05:22:24.011 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525724319}) 2024-09-11T05:22:24.011 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.012+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.012 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788489928}) 2024-09-11T05:22:24.012 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430474381}) 2024-09-11T05:22:24.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863743830}) 2024-09-11T05:22:24.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315575829}) 2024-09-11T05:22:24.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863743830}) 2024-09-11T05:22:24.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.016+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647180904}) 2024-09-11T05:22:24.016 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.016 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690301051}) 2024-09-11T05:22:24.016 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873498822}) 2024-09-11T05:22:24.016 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.016 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690301051}) 2024-09-11T05:22:24.016 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3242435743}) 2024-09-11T05:22:24.018 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.018 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179922636}) 2024-09-11T05:22:24.018 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2053202080}) 2024-09-11T05:22:24.018 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.018 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179922636}) 2024-09-11T05:22:24.019 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.020+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705165078}) 2024-09-11T05:22:24.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2700693250}) 2024-09-11T05:22:24.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=849044454}) 2024-09-11T05:22:24.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2700693250}) 2024-09-11T05:22:24.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3914111473}) 2024-09-11T05:22:24.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687564830}) 2024-09-11T05:22:24.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848576711}) 2024-09-11T05:22:24.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687564830}) 2024-09-11T05:22:24.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.024+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094444538}) 2024-09-11T05:22:24.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274239761}) 2024-09-11T05:22:24.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821179771}) 2024-09-11T05:22:24.025 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.025 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274239761}) 2024-09-11T05:22:24.025 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=978308674}) 2024-09-11T05:22:24.027 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.027 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147342007}) 2024-09-11T05:22:24.027 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358309755}) 2024-09-11T05:22:24.027 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.027 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147342007}) 2024-09-11T05:22:24.027 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.028+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2818629451}) 2024-09-11T05:22:24.029 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.029 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915046359}) 2024-09-11T05:22:24.029 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1641547336}) 2024-09-11T05:22:24.029 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.029 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915046359}) 2024-09-11T05:22:24.029 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1064614511}) 2024-09-11T05:22:24.031 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.031 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762404553}) 2024-09-11T05:22:24.031 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1317811067}) 2024-09-11T05:22:24.031 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.031 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762404553}) 2024-09-11T05:22:24.031 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.032+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3602267516}) 2024-09-11T05:22:24.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.036+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.036+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1904025959}) 2024-09-11T05:22:24.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.036+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055781248}) 2024-09-11T05:22:24.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.036+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.036+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1904025959}) 2024-09-11T05:22:24.034 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.036+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1908543833}) 2024-09-11T05:22:24.035 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:22:24.035 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:24.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288690348}) 2024-09-11T05:22:24.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101279108}) 2024-09-11T05:22:24.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288690348}) 2024-09-11T05:22:24.037 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1346169399}) 2024-09-11T05:22:24.041 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.041 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=727355193}) 2024-09-11T05:22:24.041 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280473596}) 2024-09-11T05:22:24.041 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.041 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=727355193}) 2024-09-11T05:22:24.041 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.040+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318658970}) 2024-09-11T05:22:24.041 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302142432}) 2024-09-11T05:22:24.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1868312219}) 2024-09-11T05:22:24.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302142432}) 2024-09-11T05:22:24.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136662148}) 2024-09-11T05:22:24.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1222485336}) 2024-09-11T05:22:24.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478110528}) 2024-09-11T05:22:24.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1222485336}) 2024-09-11T05:22:24.043 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.044+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359536732}) 2024-09-11T05:22:24.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.048+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.048+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558733320}) 2024-09-11T05:22:24.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.048+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553742835}) 2024-09-11T05:22:24.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.048+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.048+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558733320}) 2024-09-11T05:22:24.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.048+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613254134}) 2024-09-11T05:22:24.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.052+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.052+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071867836}) 2024-09-11T05:22:24.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.052+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=679288519}) 2024-09-11T05:22:24.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.052+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.052+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071867836}) 2024-09-11T05:22:24.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.052+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1315069844}) 2024-09-11T05:22:24.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374136445}) 2024-09-11T05:22:24.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170894280}) 2024-09-11T05:22:24.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374136445}) 2024-09-11T05:22:24.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3991422257}) 2024-09-11T05:22:24.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398785588}) 2024-09-11T05:22:24.055 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4058920473}) 2024-09-11T05:22:24.055 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.055 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398785588}) 2024-09-11T05:22:24.055 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.056+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1581761245}) 2024-09-11T05:22:24.057 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.060+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.057 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.060+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663238897}) 2024-09-11T05:22:24.057 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.060+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1092603666}) 2024-09-11T05:22:24.057 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.060+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.057 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.060+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663238897}) 2024-09-11T05:22:24.057 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.060+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266600278}) 2024-09-11T05:22:24.059 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.060+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.060+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826940485}) 2024-09-11T05:22:24.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39768534}) 2024-09-11T05:22:24.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826940485}) 2024-09-11T05:22:24.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040305065}) 2024-09-11T05:22:24.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521045838}) 2024-09-11T05:22:24.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1240943565}) 2024-09-11T05:22:24.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521045838}) 2024-09-11T05:22:24.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.064+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397712020}) 2024-09-11T05:22:24.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080805988}) 2024-09-11T05:22:24.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2631364474}) 2024-09-11T05:22:24.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080805988}) 2024-09-11T05:22:24.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2111891767}) 2024-09-11T05:22:24.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043417835}) 2024-09-11T05:22:24.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1026854350}) 2024-09-11T05:22:24.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043417835}) 2024-09-11T05:22:24.067 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.068+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829245326}) 2024-09-11T05:22:24.068 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.068 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052308373}) 2024-09-11T05:22:24.068 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957484087}) 2024-09-11T05:22:24.069 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.069 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052308373}) 2024-09-11T05:22:24.069 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1265868976}) 2024-09-11T05:22:24.070 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.070 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449018210}) 2024-09-11T05:22:24.070 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711486855}) 2024-09-11T05:22:24.070 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.071 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449018210}) 2024-09-11T05:22:24.071 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.072+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805735632}) 2024-09-11T05:22:24.072 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.072 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253952766}) 2024-09-11T05:22:24.072 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3500451489}) 2024-09-11T05:22:24.073 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.073 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253952766}) 2024-09-11T05:22:24.073 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2108003390}) 2024-09-11T05:22:24.074 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.074 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511374658}) 2024-09-11T05:22:24.074 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3103921213}) 2024-09-11T05:22:24.075 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.075 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511374658}) 2024-09-11T05:22:24.075 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.076+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011198398}) 2024-09-11T05:22:24.076 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791173235}) 2024-09-11T05:22:24.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=87429475}) 2024-09-11T05:22:24.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791173235}) 2024-09-11T05:22:24.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4066917}) 2024-09-11T05:22:24.078 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.078 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143560874}) 2024-09-11T05:22:24.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1572113547}) 2024-09-11T05:22:24.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143560874}) 2024-09-11T05:22:24.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.080+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2827275047}) 2024-09-11T05:22:24.080 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.080 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916724180}) 2024-09-11T05:22:24.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3227185409}) 2024-09-11T05:22:24.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916724180}) 2024-09-11T05:22:24.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461943697}) 2024-09-11T05:22:24.082 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.082 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676534681}) 2024-09-11T05:22:24.082 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2439984240}) 2024-09-11T05:22:24.082 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.083 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676534681}) 2024-09-11T05:22:24.083 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.084+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1431404261}) 2024-09-11T05:22:24.084 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.088+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.084 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.088+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895931041}) 2024-09-11T05:22:24.084 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.088+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3779905414}) 2024-09-11T05:22:24.084 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.088+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.085 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.088+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895931041}) 2024-09-11T05:22:24.085 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.088+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401587390}) 2024-09-11T05:22:24.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient: get_monmap_and_config 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient: build_initial_monmap 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient: monmap: 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:24.166522+0000 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:24.166522+0000 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:24.161 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:24.162 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:24.162 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 1 Processor -- start 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 1 -- start start 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient: init 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f275406e0e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f275409eb30 0x7f275413a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient(hunting): picked mon.noname-c con 0x7f275409eb30 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409ef40 0x7f275413ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient(hunting): picked mon.noname-b con 0x7f275409ef40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:24.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f275406e4c0 con 0x7f275406e0e0 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f275409f820 con 0x7f275409eb30 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f275409f540 con 0x7f275409ef40 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f275a6eb640 10 monclient(hunting): authenticate will time out at 1827.603104s 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409ef40 0x7f275413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409ef40 0x7f275413ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48550/0 (socket says 172.21.15.138:48550) 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 1 -- 172.21.15.138:0/1013636004 learned_addr learned my addr 172.21.15.138:0/1013636004 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient(hunting): get_auth_request con 0x7f275409ef40 auth_method 0 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f274bfff640 1 --2- 172.21.15.138:0/1013636004 >> v2:172.21.15.190:3300/0 conn(0x7f275409eb30 0x7f275413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f274bfff640 10 monclient(hunting): get_auth_request con 0x7f275409eb30 auth_method 0 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f274bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f274bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f274bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient(hunting): handle_auth_done global_id 84446 payload 274 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 1 -- 172.21.15.138:0/1013636004 >> v2:172.21.15.190:3300/0 conn(0x7f275409eb30 msgr2=0x7f275413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 1 --2- 172.21.15.138:0/1013636004 >> v2:172.21.15.190:3300/0 conn(0x7f275409eb30 0x7f275413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 1 -- 172.21.15.138:0/1013636004 >> v1:172.21.15.190:6789/0 conn(0x7f275406e0e0 legacy=0x7f275409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 1 monclient: found mon.noname-b 2024-09-11T05:22:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.164+0000 7f2758c88640 1 -- 172.21.15.138:0/1013636004 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275409fb00 con 0x7f275409ef40 2024-09-11T05:22:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2758c88640 1 -- 172.21.15.138:0/1013636004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409ef40 msgr2=0x7f275413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2758c88640 1 --2- 172.21.15.138:0/1013636004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409ef40 0x7f275413ce80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f274c009e50 tx=0x7f274c01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fc10116315a9f4b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 1 -- 172.21.15.138:0/1013636004 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c0a5670 con 0x7f275409ef40 2024-09-11T05:22:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: dump: 2024-09-11T05:22:24.168 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 1 -- 172.21.15.138:0/1013636004 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c0a5810 con 0x7f275409ef40 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 5 monclient: authenticate success, global_id 84446 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 1 -- 172.21.15.138:0/1013636004 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c0a5be0 con 0x7f275409ef40 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 4 set_mon_vals no callback set 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f27537fe640 10 monclient: dump: 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.170 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.171 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.171 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.171 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.171 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient: shutdown 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- 172.21.15.138:0/1013636004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409ef40 msgr2=0x7f275413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1013636004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409ef40 0x7f275413ce80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f274c009e50 tx=0x7f274c01c8f0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- 172.21.15.138:0/1013636004 shutdown_connections 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1013636004 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409ef40 0x7f275413ce80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1013636004 >> v2:172.21.15.190:3300/0 conn(0x7f275409eb30 0x7f275413a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- 172.21.15.138:0/1013636004 >> 172.21.15.138:0/1013636004 conn(0x7f2754063640 msgr2=0x7f2754063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- 172.21.15.138:0/1013636004 shutdown_connections 2024-09-11T05:22:24.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- 172.21.15.138:0/1013636004 wait complete. 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient: build_initial_monmap 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient: monmap: 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 librados: starting msgr at 2024-09-11T05:22:24.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 librados: starting objecter 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 Processor -- start 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- start start 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 librados: setting wanted keys 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 librados: calling monclient init 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient: init 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f275406e0e0 0x7f2754133c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient(hunting): picked mon.noname-a con 0x7f275406e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409eb30 0x7f2754134180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient(hunting): picked mon.noname-c con 0x7f275409eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f275409ef40 0x7f27541346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient(hunting): picked mon.noname-b con 0x7f275409ef40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:24.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f27540a2540 con 0x7f275406e0e0 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f27540a23c0 con 0x7f275409ef40 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f27540a26c0 con 0x7f275409eb30 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f275a6eb640 10 monclient(hunting): authenticate will time out at 1827.611404s 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409eb30 0x7f2754134180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f275406e0e0 0x7f2754133c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409eb30 0x7f2754134180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48560/0 (socket says 172.21.15.138:48560) 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 1 -- 172.21.15.138:0/1267999101 learned_addr learned my addr 172.21.15.138:0/1267999101 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 10 monclient(hunting): get_auth_request con 0x7f275409eb30 auth_method 0 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient(hunting): get_auth_request con 0x7f275406e0e0 auth_method 0 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f274bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2758c88640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f275409ef40 0x7f27541346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient(hunting): handle_auth_done global_id 114129 payload 824 2024-09-11T05:22:24.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 1 -- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409eb30 msgr2=0x7f2754134180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409eb30 0x7f2754134180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 1 -- 172.21.15.138:0/1267999101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f275409ef40 msgr2=0x7f27541346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f275409ef40 0x7f27541346c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 1 monclient: found mon.noname-a 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2753fff640 1 -- 172.21.15.138:0/1267999101 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2754137ca0 con 0x7f275406e0e0 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.172+0000 7f2758c88640 10 monclient: get_auth_request con 0x7f275409ef40 auth_method 0 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f2758c88640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f275409ef40 0x7f27541346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f274bfff640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409eb30 0x7f2754134180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f2753fff640 1 -- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f275406e0e0 msgr2=0x7f2754133c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f2753fff640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f275406e0e0 0x7f2754133c40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f274400b410 tx=0x7f274400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 client.?.objecter ms_handle_connect 0x7f275406e0e0 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 1 -- 172.21.15.138:0/1267999101 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744015030 con 0x7f275406e0e0 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: dump: 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.182 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 1 -- 172.21.15.138:0/1267999101 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2744004020 con 0x7f275406e0e0 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 5 monclient: authenticate success, global_id 114129 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 10 monclient: _renew_subs 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275409e110 con 0x7f275406e0e0 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 1 -- 172.21.15.138:0/1267999101 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744004410 con 0x7f275406e0e0 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:24.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 monclient: dump: 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 10 client.114129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:24.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27527fc640 4 set_mon_vals no callback set 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 10 monclient: _renew_subs 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27541363e0 con 0x7f275406e0e0 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 1 librados: init done 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f275409e420 con 0x7f275406e0e0 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 1 -- 172.21.15.138:0/1267999101 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f2744022060 con 0x7f275406e0e0 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 1 --2- 172.21.15.138:0/1267999101 >> v2:172.21.15.190:6800/1562782189 conn(0x7f273403cea0 0x7f273403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 1 -- 172.21.15.138:0/1267999101 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(613..613 src has 520..613) ==== 5985+0+0 (secure 0 0 0) 0x7f274404f660 con 0x7f275406e0e0 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 10 client.114129.objecter ms_dispatch 0x7f2754000ba0 osd_map(613..613 src has 520..613) 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 3 client.114129.objecter handle_osd_map got epochs [613,613] > 0 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 3 client.114129.objecter handle_osd_map decoding full epoch 613 2024-09-11T05:22:24.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.176+0000 7f27517fa640 20 client.114129.objecter dump_active .. 0 homeless 2024-09-11T05:22:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.180+0000 7f274bfff640 1 --2- 172.21.15.138:0/1267999101 >> v2:172.21.15.190:6800/1562782189 conn(0x7f273403cea0 0x7f273403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.180+0000 7f274bfff640 10 monclient: get_auth_request con 0x7f273403cea0 auth_method 0 2024-09-11T05:22:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.180+0000 7f27517fa640 10 client.114129.objecter ms_handle_connect 0x7f273403cea0 2024-09-11T05:22:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.180+0000 7f274bfff640 1 --2- 172.21.15.138:0/1267999101 >> v2:172.21.15.190:6800/1562782189 conn(0x7f273403cea0 0x7f273403f360 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7f2754136690 tx=0x7f273c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.184+0000 7f27517fa640 1 -- 172.21.15.138:0/1267999101 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f2744003dc0 con 0x7f275406e0e0 2024-09-11T05:22:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.184+0000 7f27517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:24.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.184+0000 7f27517fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:24.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.420+0000 7f275a6eb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:24.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.420+0000 7f275a6eb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:24.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.420+0000 7f275a6eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.420+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f275409dea0 con 0x7f275406e0e0 2024-09-11T05:22:24.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f27517fa640 1 -- 172.21.15.138:0/1267999101 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f2744003980 con 0x7f275406e0e0 2024-09-11T05:22:24.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f27517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:24.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f27517fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:24.420 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:24.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 10 librados: watch_flush enter 2024-09-11T05:22:24.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 10 librados: watch_flush exit 2024-09-11T05:22:24.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 20 client.114129.objecter shutdown clearing up homeless session... 2024-09-11T05:22:24.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 10 client.114129.objecter successfully canceled tick 2024-09-11T05:22:24.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 >> v2:172.21.15.190:6800/1562782189 conn(0x7f273403cea0 msgr2=0x7f273403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1267999101 >> v2:172.21.15.190:6800/1562782189 conn(0x7f273403cea0 0x7f273403f360 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7f2754136690 tx=0x7f273c008040 comp rx=0 tx=0).stop 2024-09-11T05:22:24.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 10 monclient: shutdown 2024-09-11T05:22:24.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:24.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f275406e0e0 msgr2=0x7f2754133c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f275406e0e0 0x7f2754133c40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f274400b410 tx=0x7f274400b8d0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 shutdown_connections 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1267999101 >> v2:172.21.15.190:6800/1562782189 conn(0x7f273403cea0 0x7f273403f360 unknown :-1 s=CLOSED pgs=3481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f275409ef40 0x7f27541346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f275409eb30 0x7f2754134180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 --2- 172.21.15.138:0/1267999101 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f275406e0e0 0x7f2754133c40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 >> 172.21.15.138:0/1267999101 conn(0x7f2754063640 msgr2=0x7f27541318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 shutdown_connections 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 -- 172.21.15.138:0/1267999101 wait complete. 2024-09-11T05:22:24.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.424+0000 7f275a6eb640 1 librados: shutdown 2024-09-11T05:22:24.438 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:22:24.439 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:22:24.439 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:22:24.439 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:22:24.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient: get_monmap_and_config 2024-09-11T05:22:24.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient: build_initial_monmap 2024-09-11T05:22:24.601 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient: monmap: 2024-09-11T05:22:24.602 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:24.602 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.602 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:24.606835+0000 2024-09-11T05:22:24.602 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:24.606835+0000 2024-09-11T05:22:24.602 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:24.603 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.603 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:24.603 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:24.603 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:24.603 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:24.603 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:24.604 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 1 Processor -- start 2024-09-11T05:22:24.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 1 -- start start 2024-09-11T05:22:24.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient: init 2024-09-11T05:22:24.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-11T05:22:24.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f246c070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:24.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f246c09e6e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:24.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f246c09eb10 0x7f246c140d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient(hunting): picked mon.noname-c con 0x7f246c09eb10 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:24.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f246c135b60 con 0x7f246c09e6e0 2024-09-11T05:22:24.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f246c0a8990 con 0x7f246c070010 2024-09-11T05:22:24.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f246c0703f0 con 0x7f246c09eb10 2024-09-11T05:22:24.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f24734bd640 10 monclient(hunting): authenticate will time out at 1828.043343s 2024-09-11T05:22:24.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f2470a58640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f246c09e6e0 0x7f246c13d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45656/0 (socket says 172.21.15.138:45656) 2024-09-11T05:22:24.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f2470a58640 1 -- 172.21.15.138:0/1538408622 learned_addr learned my addr 172.21.15.138:0/1538408622 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:22:24.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f2471a5a640 1 --2- 172.21.15.138:0/1538408622 >> v2:172.21.15.190:3300/0 conn(0x7f246c09eb10 0x7f246c140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f2471a5a640 10 monclient(hunting): get_auth_request con 0x7f246c09eb10 auth_method 0 2024-09-11T05:22:24.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f2471a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f2471a5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.604+0000 7f2471a5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 1 -- 172.21.15.138:0/1538408622 >> v1:172.21.15.138:6789/0 conn(0x7f246c09e6e0 legacy=0x7f246c13d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945910524 0 0) 0x7f246c135b60 con 0x7f246c09e6e0 2024-09-11T05:22:24.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): _negotiate 2024-09-11T05:22:24.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): my global_id is 114132 2024-09-11T05:22:24.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2458003530 con 0x7f246c09e6e0 2024-09-11T05:22:24.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 1 -- 172.21.15.138:0/1538408622 >> v1:172.21.15.190:6789/0 conn(0x7f246c070010 legacy=0x7f246c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471a5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1419037540 0 0) 0x7f2458003530 con 0x7f246c09e6e0 2024-09-11T05:22:24.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 >> v2:172.21.15.190:3300/0 conn(0x7f246c09eb10 msgr2=0x7f246c140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 --2- 172.21.15.138:0/1538408622 >> v2:172.21.15.190:3300/0 conn(0x7f246c09eb10 0x7f246c140d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 >> v1:172.21.15.190:6789/0 conn(0x7f246c070010 legacy=0x7f246c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:22:24.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:22:24.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246c09cbf0 con 0x7f246c09e6e0 2024-09-11T05:22:24.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f24680019b0 con 0x7f246c09e6e0 2024-09-11T05:22:24.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:22:24.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: dump: 2024-09-11T05:22:24.637 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.637 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.637 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.637 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.638 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.638 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.638 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.638 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.638 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.639 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:22:24.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 >> v1:172.21.15.138:6789/0 conn(0x7f246c09e6e0 legacy=0x7f246c13d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:22:24.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2458003b30 0x7f2458003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): picked mon.a con 0x7f2458003b30 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2458004810 0x7f2458004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): picked mon.c con 0x7f2458004810 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:24.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580053d0 0x7f245800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): picked mon.b con 0x7f24580053d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f246c0703f0 con 0x7f2458003b30 2024-09-11T05:22:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f245800c310 con 0x7f24580053d0 2024-09-11T05:22:24.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f245800c510 con 0x7f2458004810 2024-09-11T05:22:24.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.642 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2463fff640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:22:24.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2458004810 0x7f2458004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2458003b30 0x7f2458003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient(hunting): get_auth_request con 0x7f2458004810 auth_method 0 2024-09-11T05:22:24.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:24.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 10 monclient(hunting): get_auth_request con 0x7f2458003b30 auth_method 0 2024-09-11T05:22:24.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.644 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:24.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:22:24.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.645 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient(hunting): handle_auth_done global_id 114132 payload 293 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471a5a640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580053d0 0x7f245800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 1 -- 172.21.15.138:0/1538408622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580053d0 msgr2=0x7f245800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24580053d0 0x7f245800ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.646 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 1 -- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2458003b30 msgr2=0x7f2458003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2458003b30 0x7f2458003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 1 monclient: found mon.c 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:24.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 1 -- 172.21.15.138:0/1538408622 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2458003530 con 0x7f2458004810 2024-09-11T05:22:24.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2470a58640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f2471a5a640 1 -- 172.21.15.138:0/1538408622 reap_dead start 2024-09-11T05:22:24.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f24734bd640 5 monclient: authenticate success, global_id 114132 2024-09-11T05:22:24.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.608+0000 7f24734bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:24.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2470a58640 1 -- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2458004810 msgr2=0x7f2458004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2470a58640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2458004810 0x7f2458004c90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2468001bd0 tx=0x7f246801cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3fd421280084cd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.653 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246800c0e0 con 0x7f2458004810 2024-09-11T05:22:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 10 monclient: dump: 2024-09-11T05:22:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.654 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.655 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.655 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.655 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.655 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.655 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.656 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.656 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24680a9020 con 0x7f2458004810 2024-09-11T05:22:24.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:24.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:24.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:24.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 1 -- 172.21.15.138:0/1538408622 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680a5b40 con 0x7f2458004810 2024-09-11T05:22:24.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 4 set_mon_vals no callback set 2024-09-11T05:22:24.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:24.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f2463fff640 10 monclient: dump: 2024-09-11T05:22:24.658 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.658 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.658 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.658 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.658 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.659 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.659 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.659 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.659 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.659 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 10 monclient: shutdown 2024-09-11T05:22:24.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:24.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 1 -- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2458004810 msgr2=0x7f2458004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2458004810 0x7f2458004c90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2468001bd0 tx=0x7f246801cc90 comp rx=0 tx=0).stop 2024-09-11T05:22:24.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 1 -- 172.21.15.138:0/1538408622 shutdown_connections 2024-09-11T05:22:24.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 1 --2- 172.21.15.138:0/1538408622 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2458004810 0x7f2458004c90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 1 -- 172.21.15.138:0/1538408622 >> 172.21.15.138:0/1538408622 conn(0x7f246c0ab830 msgr2=0x7f246c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:24.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 1 -- 172.21.15.138:0/1538408622 shutdown_connections 2024-09-11T05:22:24.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.612+0000 7f24734bd640 1 -- 172.21.15.138:0/1538408622 wait complete. 2024-09-11T05:22:24.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient: build_initial_monmap 2024-09-11T05:22:24.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient: monmap: 2024-09-11T05:22:24.662 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:24.662 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:24.662 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:24.662 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:24.662 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:24.663 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.663 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:24.663 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:24.663 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:24.663 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 librados: starting msgr at 2024-09-11T05:22:24.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 librados: starting objecter 2024-09-11T05:22:24.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 Processor -- start 2024-09-11T05:22:24.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 -- start start 2024-09-11T05:22:24.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 librados: setting wanted keys 2024-09-11T05:22:24.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 librados: calling monclient init 2024-09-11T05:22:24.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient: init 2024-09-11T05:22:24.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:22:24.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2458004810 0x7f246c1d0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient(hunting): picked mon.noname-b con 0x7f2458004810 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:24.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c122a00 0x7f246c1d0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient(hunting): picked mon.noname-a con 0x7f246c122a00 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:24.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f246c1d0ec0 0x7f246c121790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient(hunting): picked mon.noname-c con 0x7f246c1d0ec0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:24.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f246c138c50 con 0x7f246c122a00 2024-09-11T05:22:24.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f246c138ad0 con 0x7f2458004810 2024-09-11T05:22:24.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f246c138dd0 con 0x7f246c1d0ec0 2024-09-11T05:22:24.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f24734bd640 10 monclient(hunting): authenticate will time out at 1828.053348s 2024-09-11T05:22:24.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c122a00 0x7f246c1d0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f246c1d0ec0 0x7f246c121790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c122a00 0x7f246c1d0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45634/0 (socket says 172.21.15.138:45634) 2024-09-11T05:22:24.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f246c1d0ec0 0x7f246c121790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48578/0 (socket says 172.21.15.138:48578) 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 -- 172.21.15.138:0/2490753012 learned_addr learned my addr 172.21.15.138:0/2490753012 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient(hunting): get_auth_request con 0x7f246c122a00 auth_method 0 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 10 monclient(hunting): get_auth_request con 0x7f246c1d0ec0 auth_method 0 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471259640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2458004810 0x7f246c1d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471259640 10 monclient(hunting): get_auth_request con 0x7f2458004810 auth_method 0 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471259640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471259640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471259640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient(hunting): handle_auth_done global_id 114135 payload 824 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f246c1d0ec0 msgr2=0x7f246c121790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f246c1d0ec0 0x7f246c121790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2458004810 msgr2=0x7f246c1d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2458004810 0x7f246c1d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 monclient: found mon.noname-a 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2470a58640 1 -- 172.21.15.138:0/2490753012 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246c0aacc0 con 0x7f246c122a00 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.616+0000 7f2471a5a640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f246c1d0ec0 0x7f246c121790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2470a58640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c122a00 msgr2=0x7f246c1d0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2470a58640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c122a00 0x7f246c1d0980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f24680010f0 tx=0x7f2458005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 client.?.objecter ms_handle_connect 0x7f246c122a00 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680019b0 con 0x7f246c122a00 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: dump: 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.671 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 5 monclient: authenticate success, global_id 114135 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24680047f0 con 0x7f246c122a00 2024-09-11T05:22:24.672 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468004c00 con 0x7f246c122a00 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24637fe640 4 set_mon_vals no callback set 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 monclient: dump: 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.673 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 10 monclient: _renew_subs 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246c0a4de0 con 0x7f246c122a00 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 10 client.114135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 10 monclient: _renew_subs 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f246c124200 con 0x7f246c122a00 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 1 librados: init done 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c121cd0 0x7f246c1220b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f246c135a70 con 0x7f246c121cd0 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c121cd0 0x7f246c1220b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f24680b6020 con 0x7f246c122a00 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 10 monclient: get_auth_request con 0x7f246c121cd0 auth_method 0 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 1 --2- 172.21.15.138:0/2490753012 >> v2:172.21.15.190:6800/1562782189 conn(0x7f245003cea0 0x7f245003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 10 monclient(hunting): handle_auth_done global_id 114138 payload 824 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 1 -- 172.21.15.138:0/2490753012 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f246c1225f0 con 0x7f246c121cd0 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(613..613 src has 520..613) ==== 5985+0+0 (secure 0 0 0) 0x7f24680e8f90 con 0x7f246c122a00 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 10 client.114135.objecter ms_dispatch 0x7f246c000ba0 osd_map(613..613 src has 520..613) 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 3 client.114135.objecter handle_osd_map got epochs [613,613] > 0 2024-09-11T05:22:24.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 3 client.114135.objecter handle_osd_map decoding full epoch 613 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2461ffb640 20 client.114135.objecter dump_active .. 0 homeless 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471a5a640 1 --2- 172.21.15.138:0/2490753012 >> v2:172.21.15.190:6800/1562782189 conn(0x7f245003cea0 0x7f245003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471a5a640 10 monclient: get_auth_request con 0x7f245003cea0 auth_method 0 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 10 client.114135.objecter ms_handle_connect 0x7f246c121cd0 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.620+0000 7f2471259640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c121cd0 msgr2=0x7f246c1220b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 10 client.114135.objecter resend_mon_ops 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2471259640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c121cd0 0x7f246c1220b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f246c052890 tx=0x7f245c0019d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c0147f0 con 0x7f246c121cd0 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 10 monclient: dump: 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.676 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f245c01a070 con 0x7f246c121cd0 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c121cd0 msgr2=0x7f246c1220b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c121cd0 0x7f246c1220b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f246c052890 tx=0x7f245c0019d0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c121cd0 msgr2=0x7f246c1220b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:24.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c121cd0 0x7f246c1220b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2471a5a640 1 --2- 172.21.15.138:0/2490753012 >> v2:172.21.15.190:6800/1562782189 conn(0x7f245003cea0 0x7f245003f360 secure :-1 s=READY pgs=3482 cs=0 l=1 rev1=1 crypto rx=0x7f246c0a8960 tx=0x7f246400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.624+0000 7f2461ffb640 10 client.114135.objecter ms_handle_connect 0x7f245003cea0 2024-09-11T05:22:24.678 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.678 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=9,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=930298467,mdsmap=1754867906,mgr=2388747459,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=3621271562,mon_config_key=2112823436,monmap=2552663431,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=2800915053,osd_snap=146755435}) 2024-09-11T05:22:24.679 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=28} crc {logm=2108709601,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=838713126,mon_config_key=866245988,monmap=967756510,nvmeofgw=1596038039,osd_metadata=3824196910,osd_pg_creating=2800915053,osd_snap=2125816019}) 2024-09-11T05:22:24.679 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.679 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=9,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=930298467,mdsmap=1754867906,mgr=2388747459,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=3621271562,mon_config_key=2112823436,monmap=2552663431,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=2800915053,osd_snap=146755435}) 2024-09-11T05:22:24.679 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.628+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26} crc {logm=727058846,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2038011284,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=2800915053,osd_snap=3513517815}) 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f24734bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f24734bd640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c09eff0 0x7f246c09f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f24734bd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f246c069fa0 con 0x7f246c09eff0 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c09eff0 0x7f246c09f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 10 monclient: get_auth_request con 0x7f246c09eff0 auth_method 0 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 10 monclient(hunting): handle_auth_done global_id 114141 payload 824 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.628+0000 7f2471259640 1 -- 172.21.15.138:0/2490753012 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f246c0a4590 con 0x7f246c09eff0 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 10 client.114135.objecter ms_handle_connect 0x7f246c09eff0 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 10 client.114135.objecter resend_mon_ops 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2471259640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c09eff0 msgr2=0x7f246c09f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2471259640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c09eff0 0x7f246c09f400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f245c019000 tx=0x7f245c0014c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c015680 con 0x7f246c09eff0 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 10 monclient: dump: 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.682 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f245c015680 con 0x7f246c09eff0 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c09eff0 msgr2=0x7f246c09f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c09eff0 0x7f246c09f400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f245c019000 tx=0x7f245c0014c0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c09eff0 msgr2=0x7f246c09f400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:24.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2461ffb640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c09eff0 0x7f246c09f400 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f245c019000 tx=0x7f245c0014c0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 58, 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 16671, 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:22:24.684 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 3, 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:24.685 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:24.686 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:22:24.687 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:24.688 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:24.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:22:24.690 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.692 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.693 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:24.694 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 2 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:22:24.695 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:22:24.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 10 librados: watch_flush enter 2024-09-11T05:22:24.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 10 librados: watch_flush exit 2024-09-11T05:22:24.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 20 client.114135.objecter shutdown clearing up homeless session... 2024-09-11T05:22:24.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 10 client.114135.objecter successfully canceled tick 2024-09-11T05:22:24.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 >> v2:172.21.15.190:6800/1562782189 conn(0x7f245003cea0 msgr2=0x7f245003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 --2- 172.21.15.138:0/2490753012 >> v2:172.21.15.190:6800/1562782189 conn(0x7f245003cea0 0x7f245003f360 secure :-1 s=READY pgs=3482 cs=0 l=1 rev1=1 crypto rx=0x7f246c0a8960 tx=0x7f246400b040 comp rx=0 tx=0).stop 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 10 monclient: shutdown 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c122a00 msgr2=0x7f246c1d0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c122a00 0x7f246c1d0980 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f24680010f0 tx=0x7f2458005840 comp rx=0 tx=0).stop 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f2471a5a640 1 -- 172.21.15.138:0/2490753012 reap_dead start 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 shutdown_connections 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 --2- 172.21.15.138:0/2490753012 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f246c122a00 0x7f246c1d0980 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 >> 172.21.15.138:0/2490753012 conn(0x7f246c0ab830 msgr2=0x7f246c0aa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 shutdown_connections 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 -- 172.21.15.138:0/2490753012 wait complete. 2024-09-11T05:22:24.698 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.632+0000 7f24734bd640 1 librados: shutdown 2024-09-11T05:22:24.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:22:24.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:22:24.699 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:22:24.700 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.652+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.700 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.652+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=3167090536,osdmap=3722189842}) 2024-09-11T05:22:24.700 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.652+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=5,osdmap=95} crc {osd_snap=3900808880,osdmap=3342879698}) 2024-09-11T05:22:24.700 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.652+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.700 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.652+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=3167090536,osdmap=3722189842}) 2024-09-11T05:22:24.701 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.652+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=3167090536,osdmap=1446025932}) 2024-09-11T05:22:24.701 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.664+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.701 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.664+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=58} crc {osdmap=2085355883}) 2024-09-11T05:22:24.701 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.664+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=3234810075}) 2024-09-11T05:22:24.701 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.664+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:22:24.701 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.664+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=58} crc {osdmap=2085355883}) 2024-09-11T05:22:24.701 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:22:24.664+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=56} crc {osdmap=1161381890}) 2024-09-11T05:22:24.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient: get_monmap_and_config 2024-09-11T05:22:24.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient: build_initial_monmap 2024-09-11T05:22:24.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient: monmap: 2024-09-11T05:22:24.816 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:24.816 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.816 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:24.821998+0000 2024-09-11T05:22:24.816 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:24.821998+0000 2024-09-11T05:22:24.817 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:24.817 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.817 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:24.817 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:24.817 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:24.817 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:24.817 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:24.817 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 1 Processor -- start 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 1 -- start start 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient: init 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fb0d00a32c0 0x7fb0d009e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d00a32c0 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0d009e960 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb0d009ed80 0x7fb0d0135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d009ed80 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d00a1b50 con 0x7fb0d009e960 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d00a1920 con 0x7fb0d00a32c0 2024-09-11T05:22:24.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d00a36a0 con 0x7fb0d009ed80 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d6b56640 10 monclient(hunting): authenticate will time out at 1828.258650s 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fb0d00a32c0 0x7fb0d009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fb0d00a32c0 0x7fb0d009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45666/0 (socket says 172.21.15.138:45666) 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 1 -- 172.21.15.138:0/1353170120 learned_addr learned my addr 172.21.15.138:0/1353170120 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient(hunting): get_auth_request con 0x7fb0d00a32c0 auth_method 0 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_done global_id 114144 payload 274 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 1 -- 172.21.15.138:0/1353170120 >> v2:172.21.15.190:3300/0 conn(0x7fb0d009ed80 msgr2=0x7fb0d0135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 1 --2- 172.21.15.138:0/1353170120 >> v2:172.21.15.190:3300/0 conn(0x7fb0d009ed80 0x7fb0d0135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 1 -- 172.21.15.138:0/1353170120 >> v1:172.21.15.190:6789/0 conn(0x7fb0d009e960 legacy=0x7fb0d01356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 1 monclient: found mon.noname-a 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.820+0000 7fb0d48f2640 1 -- 172.21.15.138:0/1353170120 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d009f560 con 0x7fb0d00a32c0 2024-09-11T05:22:24.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0d48f2640 1 -- 172.21.15.138:0/1353170120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d00a32c0 msgr2=0x7fb0d009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0d48f2640 1 --2- 172.21.15.138:0/1353170120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d00a32c0 0x7fb0d009e420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00a2000 tx=0x7fb0c401ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de25935ef149aad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 1 -- 172.21.15.138:0/1353170120 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40b4070 con 0x7fb0d00a32c0 2024-09-11T05:22:24.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:24.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: dump: 2024-09-11T05:22:24.827 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.827 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 1 -- 172.21.15.138:0/1353170120 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c40a7ad0 con 0x7fb0d00a32c0 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:24.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 1 -- 172.21.15.138:0/1353170120 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40af440 con 0x7fb0d00a32c0 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0cf7fe640 10 monclient: dump: 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0d6b56640 5 monclient: authenticate success, global_id 114144 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0d6b56640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.824+0000 7fb0d6b56640 4 set_mon_vals no callback set 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient: shutdown 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- 172.21.15.138:0/1353170120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d00a32c0 msgr2=0x7fb0d009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 --2- 172.21.15.138:0/1353170120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d00a32c0 0x7fb0d009e420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00a2000 tx=0x7fb0c401ea80 comp rx=0 tx=0).stop 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- 172.21.15.138:0/1353170120 shutdown_connections 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 --2- 172.21.15.138:0/1353170120 >> v2:172.21.15.190:3300/0 conn(0x7fb0d009ed80 0x7fb0d0135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 --2- 172.21.15.138:0/1353170120 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d00a32c0 0x7fb0d009e420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- 172.21.15.138:0/1353170120 >> 172.21.15.138:0/1353170120 conn(0x7fb0d0063640 msgr2=0x7fb0d0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- 172.21.15.138:0/1353170120 shutdown_connections 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- 172.21.15.138:0/1353170120 wait complete. 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient: build_initial_monmap 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient: monmap: 2024-09-11T05:22:24.830 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 librados: starting msgr at 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 librados: starting objecter 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 Processor -- start 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- start start 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 librados: setting wanted keys 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 librados: calling monclient init 2024-09-11T05:22:24.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient: init 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d009e960 0x7fb0d0133ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d009e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d009ed80 0x7fb0d0134010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d009ed80 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d00a32c0 0x7fb0d0134550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d00a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00a12d0 con 0x7fb0d009ed80 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d009f190 con 0x7fb0d00a32c0 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d0136000 con 0x7fb0d009e960 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d6b56640 10 monclient(hunting): authenticate will time out at 1828.266111s 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d48f2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d009e960 0x7fb0d0133ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d009ed80 0x7fb0d0134010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d009ed80 0x7fb0d0134010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45676/0 (socket says 172.21.15.138:45676) 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 -- 172.21.15.138:0/4286551598 learned_addr learned my addr 172.21.15.138:0/4286551598 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient(hunting): get_auth_request con 0x7fb0d009ed80 auth_method 0 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d48f2640 10 monclient(hunting): get_auth_request con 0x7fb0d009e960 auth_method 0 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d48f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d48f2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d48f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient(hunting): handle_auth_done global_id 114147 payload 824 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d50f3640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d00a32c0 0x7fb0d0134550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d009e960 msgr2=0x7fb0d0133ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d009e960 0x7fb0d0133ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d00a32c0 msgr2=0x7fb0d0134550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d00a32c0 0x7fb0d0134550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 monclient: found mon.noname-a 2024-09-11T05:22:24.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0cffff640 1 -- 172.21.15.138:0/4286551598 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d00a6710 con 0x7fb0d009ed80 2024-09-11T05:22:24.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d50f3640 10 monclient: get_auth_request con 0x7fb0d00a32c0 auth_method 0 2024-09-11T05:22:24.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.828+0000 7fb0d50f3640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d00a32c0 0x7fb0d0134550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cffff640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d009ed80 msgr2=0x7fb0d0134010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cffff640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d009ed80 0x7fb0d0134010 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00b3e0 tx=0x7fb0bc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 client.?.objecter ms_handle_connect 0x7fb0d009ed80 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc015020 con 0x7fb0d009ed80 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: dump: 2024-09-11T05:22:24.838 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 5 monclient: authenticate success, global_id 114147 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 10 monclient: _renew_subs 2024-09-11T05:22:24.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0bc011a60 con 0x7fb0d009ed80 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d0137af0 con 0x7fb0d009ed80 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc011e50 con 0x7fb0d009ed80 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 monclient: dump: 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.840 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 10 client.114147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 10 monclient: _renew_subs 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0ceffd640 4 set_mon_vals no callback set 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d0073f60 con 0x7fb0d009ed80 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 1 librados: init done 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d0134a90 0x7fb0d0070260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d00a1450 con 0x7fb0d0134a90 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d0134a90 0x7fb0d0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fb0bc01a400 con 0x7fb0d009ed80 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 10 monclient: get_auth_request con 0x7fb0d0134a90 auth_method 0 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_done global_id 84458 payload 824 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 1 --2- 172.21.15.138:0/4286551598 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb0b403d070 0x7fb0b403f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d48f2640 1 -- 172.21.15.138:0/4286551598 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0d009e420 con 0x7fb0d0134a90 2024-09-11T05:22:24.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(613..613 src has 520..613) ==== 5985+0+0 (secure 0 0 0) 0x7fb0bc04fdb0 con 0x7fb0d009ed80 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 10 client.114147.objecter ms_dispatch 0x7fb0d0000ba0 osd_map(613..613 src has 520..613) 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 3 client.114147.objecter handle_osd_map got epochs [613,613] > 0 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 3 client.114147.objecter handle_osd_map decoding full epoch 613 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0cd7fa640 20 client.114147.objecter dump_active .. 0 homeless 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d50f3640 1 --2- 172.21.15.138:0/4286551598 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb0b403d070 0x7fb0b403f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.832+0000 7fb0d50f3640 10 monclient: get_auth_request con 0x7fb0b403d070 auth_method 0 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 10 client.114147.objecter ms_handle_connect 0x7fb0d0134a90 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 10 client.114147.objecter resend_mon_ops 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0d48f2640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d0134a90 msgr2=0x7fb0d0070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0d48f2640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d0134a90 0x7fb0d0070260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40b1000 tx=0x7fb0c401b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40b4040 con 0x7fb0d0134a90 2024-09-11T05:22:24.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 10 monclient: dump: 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb0c40afc50 con 0x7fb0d0134a90 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:24.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:24.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d0134a90 msgr2=0x7fb0d0070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d0134a90 0x7fb0d0070260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40b1000 tx=0x7fb0c401b820 comp rx=0 tx=0).stop 2024-09-11T05:22:24.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d0134a90 msgr2=0x7fb0d0070260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:24.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d0134a90 0x7fb0d0070260 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40b1000 tx=0x7fb0c401b820 comp rx=0 tx=0).stop 2024-09-11T05:22:24.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0cd7fa640 10 client.114147.objecter ms_handle_connect 0x7fb0b403d070 2024-09-11T05:22:24.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.836+0000 7fb0d50f3640 1 --2- 172.21.15.138:0/4286551598 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb0b403d070 0x7fb0b403f530 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 crypto rx=0x7fb0d013cb00 tx=0x7fb0c8007360 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d6b56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:22:24.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d6b56640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d00707a0 0x7fb0d0070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d6b56640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d0134e70 con 0x7fb0d00707a0 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d00707a0 0x7fb0d0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 10 monclient: get_auth_request con 0x7fb0d00707a0 auth_method 0 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 10 monclient(hunting): handle_auth_done global_id 84461 payload 824 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.840+0000 7fb0d48f2640 1 -- 172.21.15.138:0/4286551598 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb0d006cca0 con 0x7fb0d00707a0 2024-09-11T05:22:24.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d48f2640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d00707a0 msgr2=0x7fb0d0070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 10 client.114147.objecter ms_handle_connect 0x7fb0d00707a0 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d48f2640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d00707a0 0x7fb0d0070bb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb0c401d130 tx=0x7fb0c40a9000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 10 client.114147.objecter resend_mon_ops 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40b9d50 con 0x7fb0d00707a0 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 10 monclient: dump: 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.848 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 58, 2024-09-11T05:22:24.849 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 388614, 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 3, 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:24.850 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:24.851 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:22:24.852 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:22:24.853 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.854 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.855 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.856 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:24.857 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.858 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:22:24.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:22:24.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:24.861 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:24.861 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:24.861 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:22:24.861 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:24.861 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:24.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:24.861 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:22:24.861 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fb0c40b9d50 con 0x7fb0d00707a0 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d00707a0 msgr2=0x7fb0d0070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d00707a0 0x7fb0d0070bb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb0c401d130 tx=0x7fb0c40a9000 comp rx=0 tx=0).stop 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d00707a0 msgr2=0x7fb0d0070bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0cd7fa640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb0d00707a0 0x7fb0d0070bb0 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb0c401d130 tx=0x7fb0c40a9000 comp rx=0 tx=0).stop 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 10 librados: watch_flush enter 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 10 librados: watch_flush exit 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 20 client.114147.objecter shutdown clearing up homeless session... 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 10 client.114147.objecter successfully canceled tick 2024-09-11T05:22:24.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb0b403d070 msgr2=0x7fb0b403f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 --2- 172.21.15.138:0/4286551598 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb0b403d070 0x7fb0b403f530 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 crypto rx=0x7fb0d013cb00 tx=0x7fb0c8007360 comp rx=0 tx=0).stop 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 10 monclient: shutdown 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d009ed80 msgr2=0x7fb0d0134010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d009ed80 0x7fb0d0134010 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00b3e0 tx=0x7fb0bc00b8a0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d50f3640 1 -- 172.21.15.138:0/4286551598 reap_dead start 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 shutdown_connections 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 --2- 172.21.15.138:0/4286551598 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb0d009ed80 0x7fb0d0134010 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 >> 172.21.15.138:0/4286551598 conn(0x7fb0d0063640 msgr2=0x7fb0d00a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 shutdown_connections 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 -- 172.21.15.138:0/4286551598 wait complete. 2024-09-11T05:22:24.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.844+0000 7fb0d6b56640 1 librados: shutdown 2024-09-11T05:22:24.865 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:22:24.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:22:24.865 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:22:24.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient: get_monmap_and_config 2024-09-11T05:22:24.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient: build_initial_monmap 2024-09-11T05:22:24.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient: monmap: 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:24.986260+0000 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:24.986260+0000 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:24.981 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 1 Processor -- start 2024-09-11T05:22:24.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 1 -- start start 2024-09-11T05:22:24.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient: init 2024-09-11T05:22:24.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6ba409da20 0x7f6ba40a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba409da20 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ba40a1e60 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ba40a22d0 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ba4064590 con 0x7f6ba40a22d0 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ba40a2c70 con 0x7f6ba40a1e60 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6ba409de00 con 0x7f6ba409da20 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba99e6640 10 monclient(hunting): authenticate will time out at 1828.422259s 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba37fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f6ba40a22d0 0x7f6ba4136910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45664/0 (socket says 172.21.15.138:45664) 2024-09-11T05:22:24.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba37fe640 1 -- 172.21.15.138:0/3231154917 learned_addr learned my addr 172.21.15.138:0/3231154917 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:22:24.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba2ffd640 1 --2- 172.21.15.138:0/3231154917 >> v2:172.21.15.190:3300/0 conn(0x7f6ba409da20 0x7f6ba40a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba2ffd640 10 monclient(hunting): get_auth_request con 0x7f6ba409da20 auth_method 0 2024-09-11T05:22:24.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.984+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 -- 172.21.15.138:0/3231154917 >> v1:172.21.15.138:6789/0 conn(0x7f6ba40a22d0 legacy=0x7f6ba4136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1554395463 0 0) 0x7f6ba4064590 con 0x7f6ba40a22d0 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): _negotiate 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): my global_id is 114150 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b8c003530 con 0x7f6ba40a22d0 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1061956754 0 0) 0x7f6b8c003530 con 0x7f6ba40a22d0 2024-09-11T05:22:24.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 >> v2:172.21.15.190:3300/0 conn(0x7f6ba409da20 msgr2=0x7f6ba40a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 --2- 172.21.15.138:0/3231154917 >> v2:172.21.15.190:3300/0 conn(0x7f6ba409da20 0x7f6ba40a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 >> v1:172.21.15.190:6789/0 conn(0x7f6ba40a1e60 legacy=0x7f6ba4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 1 -- 172.21.15.138:0/3231154917 >> v1:172.21.15.190:6789/0 conn(0x7f6ba40a1e60 legacy=0x7f6ba4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba40a2f50 con 0x7f6ba40a22d0 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f6b9c001d30 con 0x7f6ba40a22d0 2024-09-11T05:22:24.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: dump: 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:24.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 >> v1:172.21.15.138:6789/0 conn(0x7f6ba40a22d0 legacy=0x7f6ba4136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b8c003b30 0x7f6b8c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): picked mon.b con 0x7f6b8c003b30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6b8c004810 0x7f6b8c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): picked mon.c con 0x7f6b8c004810 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 0x7f6b8c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): picked mon.a con 0x7f6b8c0053d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba409de00 con 0x7f6b8c0053d0 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c00c310 con 0x7f6b8c003b30 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8c00c510 con 0x7f6b8c004810 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 0x7f6b8c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient(hunting): get_auth_request con 0x7f6b8c0053d0 auth_method 0 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6b8c004810 0x7f6b8c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 10 monclient(hunting): get_auth_request con 0x7f6b8c004810 auth_method 0 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba27fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba2ffd640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b8c003b30 0x7f6b8c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba2ffd640 10 monclient(hunting): get_auth_request con 0x7f6b8c003b30 auth_method 0 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:24.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient(hunting): handle_auth_done global_id 114150 payload 293 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 -- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6b8c004810 msgr2=0x7f6b8c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6b8c004810 0x7f6b8c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 -- 172.21.15.138:0/3231154917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b8c003b30 msgr2=0x7f6b8c003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b8c003b30 0x7f6b8c003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 monclient: found mon.a 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 -- 172.21.15.138:0/3231154917 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c003530 con 0x7f6b8c0053d0 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient: _finish_auth 0 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 10 monclient: _check_auth_tickets 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:24.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba99e6640 5 monclient: authenticate success, global_id 114150 2024-09-11T05:22:24.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba99e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:24.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.988+0000 7f6ba37fe640 1 -- 172.21.15.138:0/3231154917 reap_dead start 2024-09-11T05:22:24.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba37fe640 1 -- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 msgr2=0x7f6b8c00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:24.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba37fe640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 0x7f6b8c00ba30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c001860 tx=0x7f6b9c01d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d905d87eb9a0e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00a6d0 con 0x7f6b8c0053d0 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 10 monclient: dump: 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.003 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b9c0aa420 con 0x7f6b8c0053d0 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 1 -- 172.21.15.138:0/3231154917 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c0aa610 con 0x7f6b8c0053d0 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 4 set_mon_vals no callback set 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba1ffb640 10 monclient: dump: 2024-09-11T05:22:25.004 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 10 monclient: shutdown 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 -- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 msgr2=0x7f6b8c00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 0x7f6b8c00ba30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c001860 tx=0x7f6b9c01d460 comp rx=0 tx=0).stop 2024-09-11T05:22:25.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 -- 172.21.15.138:0/3231154917 shutdown_connections 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 --2- 172.21.15.138:0/3231154917 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 0x7f6b8c00ba30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 -- 172.21.15.138:0/3231154917 >> 172.21.15.138:0/3231154917 conn(0x7f6ba4063640 msgr2=0x7f6ba4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 -- 172.21.15.138:0/3231154917 shutdown_connections 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 -- 172.21.15.138:0/3231154917 wait complete. 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 10 monclient: build_initial_monmap 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 10 monclient: monmap: 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:25.006 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 librados: starting msgr at 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 librados: starting objecter 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.992+0000 7f6ba99e6640 1 Processor -- start 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 -- start start 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 librados: setting wanted keys 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 librados: calling monclient init 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient: init 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 0x7f6ba40733f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6b8c0053d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba4070db0 0x7f6ba4129af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba4070db0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ba412a030 0x7f6ba406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba412a030 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba40a1aa0 con 0x7f6b8c0053d0 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba40a1920 con 0x7f6ba4070db0 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba40a1c20 con 0x7f6ba412a030 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba99e6640 10 monclient(hunting): authenticate will time out at 1828.432608s 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ba412a030 0x7f6ba406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba2ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 0x7f6ba40733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ba412a030 0x7f6ba406f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:48622/0 (socket says 172.21.15.138:48622) 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 -- 172.21.15.138:0/1836173430 learned_addr learned my addr 172.21.15.138:0/1836173430 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient(hunting): get_auth_request con 0x7f6ba412a030 auth_method 0 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba2ffd640 10 monclient(hunting): get_auth_request con 0x7f6b8c0053d0 auth_method 0 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:25.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba27fc640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba4070db0 0x7f6ba4129af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient(hunting): handle_auth_done global_id 84464 payload 824 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba4070db0 msgr2=0x7f6ba4129af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba4070db0 0x7f6ba4129af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 msgr2=0x7f6ba40733f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6b8c0053d0 0x7f6ba40733f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 monclient: found mon.noname-c 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:25.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 -- 172.21.15.138:0/1836173430 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba412a4e0 con 0x7f6ba412a030 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba27fc640 10 monclient: get_auth_request con 0x7f6ba4070db0 auth_method 0 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba27fc640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba4070db0 0x7f6ba4129af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ba412a030 msgr2=0x7f6ba406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba37fe640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ba412a030 0x7f6ba406f5a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c01bb60 tx=0x7f6b8c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 client.?.objecter ms_handle_connect 0x7f6ba412a030 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c0b9070 con 0x7f6ba412a030 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: dump: 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.011 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: _finish_auth 0 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b9c004b70 con 0x7f6ba412a030 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c001d30 con 0x7f6ba412a030 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6b8bfff640 10 monclient: dump: 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:24.996+0000 7f6ba17fa640 4 set_mon_vals no callback set 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 5 monclient: authenticate success, global_id 84464 2024-09-11T05:22:25.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 10 monclient: _renew_subs 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba4072fd0 con 0x7f6ba412a030 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 10 client.84464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 10 monclient: _renew_subs 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba4071470 con 0x7f6ba412a030 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 1 librados: init done 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba406fae0 0x7f6ba406fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba4070400 con 0x7f6ba406fae0 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f6b9c0aa420 con 0x7f6ba412a030 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba406fae0 0x7f6ba406fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.014 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 10 monclient: get_auth_request con 0x7f6ba406fae0 auth_method 0 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6b8bfff640 1 --2- 172.21.15.138:0/1836173430 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6b9003cef0 0x7f6b9003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(614..614 src has 520..614) ==== 5985+0+0 (secure 0 0 0) 0x7f6b9c0a7020 con 0x7f6ba412a030 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6b8bfff640 10 client.84464.objecter ms_dispatch 0x7f6ba4000ba0 osd_map(614..614 src has 520..614) 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6b8bfff640 3 client.84464.objecter handle_osd_map got epochs [614,614] > 0 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6b8bfff640 3 client.84464.objecter handle_osd_map decoding full epoch 614 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6b8bfff640 20 client.84464.objecter dump_active .. 0 homeless 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba27fc640 1 --2- 172.21.15.138:0/1836173430 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6b9003cef0 0x7f6b9003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.015 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba27fc640 10 monclient: get_auth_request con 0x7f6b9003cef0 auth_method 0 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_done global_id 104191 payload 824 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.000+0000 7f6ba2ffd640 1 -- 172.21.15.138:0/1836173430 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ba4071700 con 0x7f6ba406fae0 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6ba2ffd640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba406fae0 msgr2=0x7f6ba406fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6ba2ffd640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba406fae0 0x7f6ba406fec0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6b980146f0 tx=0x7f6b98014bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 10 client.84464.objecter ms_handle_connect 0x7f6ba406fae0 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 10 client.84464.objecter resend_mon_ops 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 10 client.84464.objecter ms_handle_connect 0x7f6b9003cef0 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6ba27fc640 1 --2- 172.21.15.138:0/1836173430 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6b9003cef0 0x7f6b9003f3b0 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7f6b94000c10 tx=0x7f6b94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b98012840 con 0x7f6ba406fae0 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 10 monclient: dump: 2024-09-11T05:22:25.016 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6b98012840 con 0x7f6ba406fae0 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba406fae0 msgr2=0x7f6ba406fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.017 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba406fae0 0x7f6ba406fec0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6b980146f0 tx=0x7f6b98014bb0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba406fae0 msgr2=0x7f6ba406fec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:25.018 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.004+0000 7f6b8bfff640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba406fae0 0x7f6ba406fec0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6b980146f0 tx=0x7f6b98014bb0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.019 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.008+0000 7f6ba99e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.008+0000 7f6ba99e6640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba412b9f0 0x7f6ba412be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.008+0000 7f6ba99e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.008+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba406cca0 con 0x7f6ba412b9f0 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba412b9f0 0x7f6ba412be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 10 monclient: get_auth_request con 0x7f6ba412b9f0 auth_method 0 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:25.020 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 10 monclient(hunting): handle_auth_done global_id 104194 payload 824 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 1 -- 172.21.15.138:0/1836173430 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ba4072730 con 0x7f6ba412b9f0 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba412b9f0 msgr2=0x7f6ba412be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6b8bfff640 10 client.84464.objecter ms_handle_connect 0x7f6ba412b9f0 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6b8bfff640 10 client.84464.objecter resend_mon_ops 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6ba2ffd640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba412b9f0 0x7f6ba412be00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b980146f0 tx=0x7f6b98004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9802d020 con 0x7f6ba412b9f0 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6b8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6b8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.012+0000 7f6b8bfff640 10 monclient: dump: 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.021 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f6b98016430 con 0x7f6ba412b9f0 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6b8bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6b8bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba412b9f0 msgr2=0x7f6ba412be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6b8bfff640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba412b9f0 0x7f6ba412be00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b980146f0 tx=0x7f6b98004740 comp rx=0 tx=0).stop 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6b8bfff640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba412b9f0 msgr2=0x7f6ba412be00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:25.022 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6b8bfff640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ba412b9f0 0x7f6ba412be00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 58, 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 135175, 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:22:25.023 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 4, 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:25.024 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:25.025 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:22:25.026 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:25.027 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.028 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:25.029 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:22:25.030 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.031 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:25.032 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:25.033 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:25.034 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:25.035 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:25.035 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:25.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:25.035 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:22:25.035 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:22:25.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 10 librados: watch_flush enter 2024-09-11T05:22:25.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 10 librados: watch_flush exit 2024-09-11T05:22:25.036 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 20 client.84464.objecter shutdown clearing up homeless session... 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 10 client.84464.objecter successfully canceled tick 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6b9003cef0 msgr2=0x7f6b9003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 --2- 172.21.15.138:0/1836173430 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6b9003cef0 0x7f6b9003f3b0 secure :-1 s=READY pgs=3484 cs=0 l=1 rev1=1 crypto rx=0x7f6b94000c10 tx=0x7f6b94009040 comp rx=0 tx=0).stop 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 10 monclient: shutdown 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ba412a030 msgr2=0x7f6ba406f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 --2- 172.21.15.138:0/1836173430 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ba412a030 0x7f6ba406f5a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c01bb60 tx=0x7f6b8c005840 comp rx=0 tx=0).stop 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba37fe640 1 -- 172.21.15.138:0/1836173430 reap_dead start 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba37fe640 1 -- 172.21.15.138:0/1836173430 reap_dead start 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 shutdown_connections 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 >> 172.21.15.138:0/1836173430 conn(0x7f6ba4063640 msgr2=0x7f6ba40a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 shutdown_connections 2024-09-11T05:22:25.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 -- 172.21.15.138:0/1836173430 wait complete. 2024-09-11T05:22:25.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.016+0000 7f6ba99e6640 1 librados: shutdown 2024-09-11T05:22:25.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:22:25.038 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T05:22:25.038 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T05:22:25.038 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T05:22:25.038 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T05:22:25.039 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T05:22:25.039 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:22:25.057 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T05:22:25.057 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:22:25.057 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient: get_monmap_and_config 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient: build_initial_monmap 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient: monmap: 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:25.174839+0000 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:25.174839+0000 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:25.169 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.170 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:25.170 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:25.170 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:25.170 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:25.170 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:25.170 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 1 Processor -- start 2024-09-11T05:22:25.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 1 -- start start 2024-09-11T05:22:25.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient: init 2024-09-11T05:22:25.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:25.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-11T05:22:25.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c06e0e0 0x7f162c136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient(hunting): picked mon.noname-b con 0x7f162c06e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f162c136170 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f162c136c60 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f162c0a3270 con 0x7f162c136170 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f162c0a0af0 con 0x7f162c136c60 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f162c06e4c0 con 0x7f162c06e0e0 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1630b41640 10 monclient(hunting): authenticate will time out at 1828.611129s 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1629d9b640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f162c136170 0x7f162c0a1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:45670/0 (socket says 172.21.15.138:45670) 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f1629d9b640 1 -- 172.21.15.138:0/3722012557 learned_addr learned my addr 172.21.15.138:0/3722012557 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 1 --2- 172.21.15.138:0/3722012557 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c06e0e0 0x7f162c136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient(hunting): get_auth_request con 0x7f162c06e0e0 auth_method 0 2024-09-11T05:22:25.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162ad9d640 1 -- 172.21.15.138:0/3722012557 >> v1:172.21.15.190:6789/0 conn(0x7f162c136c60 legacy=0x7f162c0a2030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient(hunting): handle_auth_done global_id 84467 payload 274 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 1 -- 172.21.15.138:0/3722012557 >> v1:172.21.15.190:6789/0 conn(0x7f162c136c60 legacy=0x7f162c0a2030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 1 -- 172.21.15.138:0/3722012557 >> v1:172.21.15.138:6789/0 conn(0x7f162c136170 legacy=0x7f162c0a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient: _finish_hunting 0 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 1 monclient: found mon.noname-b 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:25.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.172+0000 7f162a59c640 1 -- 172.21.15.138:0/3722012557 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162c0a2740 con 0x7f162c06e0e0 2024-09-11T05:22:25.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162a59c640 1 -- 172.21.15.138:0/3722012557 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c06e0e0 msgr2=0x7f162c136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:25.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162a59c640 1 --2- 172.21.15.138:0/3722012557 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c06e0e0 0x7f162c136720 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1624009dc0 tx=0x7f162401c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ecea118642e757 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:25.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 1 -- 172.21.15.138:0/3722012557 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16240ad070 con 0x7f162c06e0e0 2024-09-11T05:22:25.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:25.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: dump: 2024-09-11T05:22:25.179 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: _finish_auth 0 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: _check_auth_tickets 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:25.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f1630b41640 5 monclient: authenticate success, global_id 84467 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f1630b41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 1 -- 172.21.15.138:0/3722012557 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16240a9750 con 0x7f162c06e0e0 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f1630b41640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 1 -- 172.21.15.138:0/3722012557 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16240a9b20 con 0x7f162c06e0e0 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f1630b41640 4 set_mon_vals no callback set 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.176+0000 7f162959a640 10 monclient: dump: 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.181 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient: shutdown 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- 172.21.15.138:0/3722012557 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c06e0e0 msgr2=0x7f162c136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 --2- 172.21.15.138:0/3722012557 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c06e0e0 0x7f162c136720 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1624009dc0 tx=0x7f162401c670 comp rx=0 tx=0).stop 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- 172.21.15.138:0/3722012557 shutdown_connections 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 --2- 172.21.15.138:0/3722012557 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c06e0e0 0x7f162c136720 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- 172.21.15.138:0/3722012557 >> 172.21.15.138:0/3722012557 conn(0x7f162c063640 msgr2=0x7f162c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:25.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- 172.21.15.138:0/3722012557 shutdown_connections 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- 172.21.15.138:0/3722012557 wait complete. 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient: build_initial_monmap 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient: monmap: 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.183 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 librados: starting msgr at 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 librados: starting objecter 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 Processor -- start 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- start start 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 librados: setting wanted keys 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 librados: calling monclient init 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient: init 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f162c06e0e0 0x7f162c1cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient(hunting): picked mon.noname-b con 0x7f162c06e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f162c136170 0x7f162c1cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient(hunting): picked mon.noname-a con 0x7f162c136170 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c136c60 0x7f162c1d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient(hunting): picked mon.noname-c con 0x7f162c136c60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f162c1420b0 con 0x7f162c136170 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f162c141f30 con 0x7f162c06e0e0 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f162c142230 con 0x7f162c136c60 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1630b41640 10 monclient(hunting): authenticate will time out at 1828.618174s 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f162c136170 0x7f162c1cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c136c60 0x7f162c1d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f162c136170 0x7f162c1cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:45696/0 (socket says 172.21.15.138:45696) 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 -- 172.21.15.138:0/2745945710 learned_addr learned my addr 172.21.15.138:0/2745945710 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:25.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient(hunting): get_auth_request con 0x7f162c136170 auth_method 0 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 10 monclient(hunting): get_auth_request con 0x7f162c136c60 auth_method 0 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162a59c640 1 -- 172.21.15.138:0/2745945710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f162c06e0e0 msgr2=0x7f162c1cd910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162a59c640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f162c06e0e0 0x7f162c1cd910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:25.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient(hunting): handle_auth_done global_id 114156 payload 824 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 -- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c136c60 msgr2=0x7f162c1d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c136c60 0x7f162c1d0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 -- 172.21.15.138:0/2745945710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f162c06e0e0 msgr2=0x7f162c1cd910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f162c06e0e0 0x7f162c1cd910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 monclient: found mon.noname-a 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f1629d9b640 1 -- 172.21.15.138:0/2745945710 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162c074070 con 0x7f162c136170 2024-09-11T05:22:25.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.180+0000 7f162ad9d640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c136c60 0x7f162c1d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1629d9b640 1 -- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f162c136170 msgr2=0x7f162c1cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1629d9b640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f162c136170 0x7f162c1cde50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c480 tx=0x7f161c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 client.?.objecter ms_handle_connect 0x7f162c136170 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 1 -- 172.21.15.138:0/2745945710 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c007070 con 0x7f162c136170 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: dump: 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.190 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: _finish_auth 0 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 1 -- 172.21.15.138:0/2745945710 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f161c007210 con 0x7f162c136170 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 1 -- 172.21.15.138:0/2745945710 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c007600 con 0x7f162c136170 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:25.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 monclient: dump: 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 5 monclient: authenticate success, global_id 114156 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 10 monclient: _renew_subs 2024-09-11T05:22:25.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1628d99640 4 set_mon_vals no callback set 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f162c0a66a0 con 0x7f162c136170 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 10 client.114156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 10 monclient: _renew_subs 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f162c09e110 con 0x7f162c136170 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 1 librados: init done 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 1 -- 172.21.15.138:0/2745945710 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f161c01b400 con 0x7f162c136170 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f162c136720 con 0x7f162c136170 2024-09-11T05:22:25.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 1 --2- 172.21.15.138:0/2745945710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f161003ce50 0x7f161003f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 1 -- 172.21.15.138:0/2745945710 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(614..614 src has 520..614) ==== 5985+0+0 (secure 0 0 0) 0x7f161c04f980 con 0x7f162c136170 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 10 client.114156.objecter ms_dispatch 0x7f162c000ba0 osd_map(614..614 src has 520..614) 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 3 client.114156.objecter handle_osd_map got epochs [614,614] > 0 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 3 client.114156.objecter handle_osd_map decoding full epoch 614 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f1622ffd640 20 client.114156.objecter dump_active .. 0 homeless 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f162a59c640 1 --2- 172.21.15.138:0/2745945710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f161003ce50 0x7f161003f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.184+0000 7f162a59c640 10 monclient: get_auth_request con 0x7f161003ce50 auth_method 0 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.188+0000 7f1622ffd640 10 client.114156.objecter ms_handle_connect 0x7f161003ce50 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.188+0000 7f162a59c640 1 --2- 172.21.15.138:0/2745945710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f161003ce50 0x7f161003f310 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7f162c0a4dd0 tx=0x7f16240a7000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.188+0000 7f1622ffd640 1 -- 172.21.15.138:0/2745945710 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f161c01ac60 con 0x7f162c136170 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.188+0000 7f1622ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:25.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.188+0000 7f1622ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:25.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:25.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:25.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:25.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f162c12c690 con 0x7f162c136170 2024-09-11T05:22:25.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1622ffd640 1 -- 172.21.15.138:0/2745945710 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f161c04dec0 con 0x7f162c136170 2024-09-11T05:22:25.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1622ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:25.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1622ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:25.421 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:25.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 10 librados: watch_flush enter 2024-09-11T05:22:25.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 10 librados: watch_flush exit 2024-09-11T05:22:25.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 20 client.114156.objecter shutdown clearing up homeless session... 2024-09-11T05:22:25.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 10 client.114156.objecter successfully canceled tick 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f161003ce50 msgr2=0x7f161003f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 --2- 172.21.15.138:0/2745945710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f161003ce50 0x7f161003f310 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7f162c0a4dd0 tx=0x7f16240a7000 comp rx=0 tx=0).stop 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 10 monclient: shutdown 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f162c136170 msgr2=0x7f162c1cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f162c136170 0x7f162c1cde50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c480 tx=0x7f161c00c940 comp rx=0 tx=0).stop 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 shutdown_connections 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 --2- 172.21.15.138:0/2745945710 >> v2:172.21.15.190:6800/1562782189 conn(0x7f161003ce50 0x7f161003f310 unknown :-1 s=CLOSED pgs=3485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f162c136c60 0x7f162c1d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f162c136170 0x7f162c1cde50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 --2- 172.21.15.138:0/2745945710 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f162c06e0e0 0x7f162c1cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 >> 172.21.15.138:0/2745945710 conn(0x7f162c063640 msgr2=0x7f162c12b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:25.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 shutdown_connections 2024-09-11T05:22:25.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 -- 172.21.15.138:0/2745945710 wait complete. 2024-09-11T05:22:25.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:25.424+0000 7f1630b41640 1 librados: shutdown 2024-09-11T05:22:28.441 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient: get_monmap_and_config 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient: build_initial_monmap 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient: monmap: 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:28.578756+0000 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:28.578756+0000 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:28.573 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:28.574 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:28.574 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:28.574 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:28.574 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 1 Processor -- start 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 1 -- start start 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient: init 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f265c06e0e0 0x7f265c136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient(hunting): picked mon.noname-a con 0x7f265c06e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f265c136170 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f265c136c60 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:28.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265c0a3270 con 0x7f265c136170 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265c0a0af0 con 0x7f265c136c60 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f265c137070 con 0x7f265c06e0e0 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f26627c6640 10 monclient(hunting): authenticate will time out at 1832.015412s 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f265c06e0e0 0x7f265c136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f2660d63640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7f265c136c60 legacy=0x7f265c0a2030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f265c06e0e0 0x7f265c136720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:38416/0 (socket says 172.21.15.138:38416) 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f2653fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f265c136170 0x7f265c0a1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50360/0 (socket says 172.21.15.138:50360) 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 1 -- 172.21.15.138:0/1788235958 learned_addr learned my addr 172.21.15.138:0/1788235958 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient(hunting): get_auth_request con 0x7f265c06e0e0 auth_method 0 2024-09-11T05:22:28.577 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient(hunting): handle_auth_done global_id 114159 payload 274 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 1 -- 172.21.15.138:0/1788235958 >> v1:172.21.15.190:6789/0 conn(0x7f265c136c60 legacy=0x7f265c0a2030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 1 -- 172.21.15.138:0/1788235958 >> v1:172.21.15.138:6789/0 conn(0x7f265c136170 legacy=0x7f265c0a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 1 monclient: found mon.noname-a 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:28.578 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.575+0000 7f265bfff640 1 -- 172.21.15.138:0/1788235958 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265c0a2740 con 0x7f265c06e0e0 2024-09-11T05:22:28.580 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265bfff640 1 -- 172.21.15.138:0/1788235958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c06e0e0 msgr2=0x7f265c136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265bfff640 1 --2- 172.21.15.138:0/1788235958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c06e0e0 0x7f265c136720 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f264c009870 tx=0x7f264c01c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8099f808767019c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 1 -- 172.21.15.138:0/1788235958 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c0b0070 con 0x7f265c06e0e0 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: dump: 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:28.581 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 5 monclient: authenticate success, global_id 114159 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 1 -- 172.21.15.138:0/1788235958 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f264c0ab600 con 0x7f265c06e0e0 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:28.582 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 4 set_mon_vals no callback set 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 1 -- 172.21.15.138:0/1788235958 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c0ab9d0 con 0x7f265c06e0e0 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265b7fe640 10 monclient: dump: 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:28.583 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient: shutdown 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- 172.21.15.138:0/1788235958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c06e0e0 msgr2=0x7f265c136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 --2- 172.21.15.138:0/1788235958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c06e0e0 0x7f265c136720 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f264c009870 tx=0x7f264c01c240 comp rx=0 tx=0).stop 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- 172.21.15.138:0/1788235958 shutdown_connections 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 --2- 172.21.15.138:0/1788235958 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c06e0e0 0x7f265c136720 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- 172.21.15.138:0/1788235958 >> 172.21.15.138:0/1788235958 conn(0x7f265c063640 msgr2=0x7f265c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- 172.21.15.138:0/1788235958 shutdown_connections 2024-09-11T05:22:28.584 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- 172.21.15.138:0/1788235958 wait complete. 2024-09-11T05:22:28.588 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient: build_initial_monmap 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient: monmap: 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:28.589 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 librados: starting msgr at 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 librados: starting objecter 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 Processor -- start 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- start start 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 librados: setting wanted keys 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 librados: calling monclient init 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient: init 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f265c136170 0x7f265c1ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:28.590 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient(hunting): picked mon.noname-b con 0x7f265c136170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f265c136c60 0x7f265c13e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient(hunting): picked mon.noname-c con 0x7f265c136c60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c13ec70 0x7f265c0708d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient(hunting): picked mon.noname-a con 0x7f265c13ec70 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f265c1420b0 con 0x7f265c13ec70 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f265c141f30 con 0x7f265c136170 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f265c142230 con 0x7f265c136c60 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:28.591 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f26627c6640 10 monclient(hunting): authenticate will time out at 1832.022642s 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f265c136c60 0x7f265c13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265bfff640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f265c136170 msgr2=0x7f265c1ce8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f265c136c60 0x7f265c13e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55966/0 (socket says 172.21.15.138:55966) 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c13ec70 0x7f265c0708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 1 -- 172.21.15.138:0/4066636887 learned_addr learned my addr 172.21.15.138:0/4066636887 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 10 monclient(hunting): get_auth_request con 0x7f265c136c60 auth_method 0 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f265bfff640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f265c136170 0x7f265c1ce8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient(hunting): get_auth_request con 0x7f265c13ec70 auth_method 0 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:28.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient(hunting): handle_auth_done global_id 114162 payload 824 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 1 -- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f265c136c60 msgr2=0x7f265c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f265c136c60 0x7f265c13e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 1 -- 172.21.15.138:0/4066636887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f265c136170 msgr2=0x7f265c1ce8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:28.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f265c136170 0x7f265c1ce8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient: _finish_hunting 0 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 1 monclient: found mon.noname-a 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2660d63640 1 -- 172.21.15.138:0/4066636887 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265c133f70 con 0x7f265c13ec70 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.583+0000 7f2653fff640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f265c136c60 0x7f265c13e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f2660d63640 1 -- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c13ec70 msgr2=0x7f265c0708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f2660d63640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c13ec70 0x7f265c0708d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f265400ba10 tx=0x7f265400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 client.?.objecter ms_handle_connect 0x7f265c13ec70 2024-09-11T05:22:28.594 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 1 -- 172.21.15.138:0/4066636887 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265400ccd0 con 0x7f265c13ec70 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: dump: 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:28.595 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: _finish_auth 0 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: _check_auth_tickets 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 1 -- 172.21.15.138:0/4066636887 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2654010070 con 0x7f265c13ec70 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 5 monclient: authenticate success, global_id 114162 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 1 -- 172.21.15.138:0/4066636887 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26540156b0 con 0x7f265c13ec70 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 monclient: dump: 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:28.596 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 10 monclient: _renew_subs 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265c135d20 con 0x7f265c13ec70 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f265a7fc640 4 set_mon_vals no callback set 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 10 client.114162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:28.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 10 monclient: _renew_subs 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f265c074270 con 0x7f265c13ec70 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 1 librados: init done 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f265c072db0 con 0x7f265c13ec70 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 1 -- 172.21.15.138:0/4066636887 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f265401d030 con 0x7f265c13ec70 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 1 --2- 172.21.15.138:0/4066636887 >> v2:172.21.15.190:6800/1562782189 conn(0x7f263c03cea0 0x7f263c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 1 -- 172.21.15.138:0/4066636887 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(614..614 src has 520..614) ==== 5985+0+0 (secure 0 0 0) 0x7f265404f910 con 0x7f265c13ec70 2024-09-11T05:22:28.598 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 10 client.114162.objecter ms_dispatch 0x7f265c000ba0 osd_map(614..614 src has 520..614) 2024-09-11T05:22:28.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 3 client.114162.objecter handle_osd_map got epochs [614,614] > 0 2024-09-11T05:22:28.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 3 client.114162.objecter handle_osd_map decoding full epoch 614 2024-09-11T05:22:28.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f26597fa640 20 client.114162.objecter dump_active .. 0 homeless 2024-09-11T05:22:28.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f265bfff640 1 --2- 172.21.15.138:0/4066636887 >> v2:172.21.15.190:6800/1562782189 conn(0x7f263c03cea0 0x7f263c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:28.599 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.587+0000 7f265bfff640 10 monclient: get_auth_request con 0x7f263c03cea0 auth_method 0 2024-09-11T05:22:28.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.591+0000 7f265bfff640 1 --2- 172.21.15.138:0/4066636887 >> v2:172.21.15.190:6800/1562782189 conn(0x7f263c03cea0 0x7f263c03f360 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f265c130830 tx=0x7f264c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:28.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.591+0000 7f26597fa640 10 client.114162.objecter ms_handle_connect 0x7f263c03cea0 2024-09-11T05:22:28.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.591+0000 7f26597fa640 1 -- 172.21.15.138:0/4066636887 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f2654012d30 con 0x7f265c13ec70 2024-09-11T05:22:28.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.591+0000 7f26597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:28.600 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.591+0000 7f26597fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:28.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.819+0000 7f26627c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:28.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.819+0000 7f26627c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:28.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.819+0000 7f26627c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:28.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.819+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f265c136720 con 0x7f265c13ec70 2024-09-11T05:22:28.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26597fa640 1 -- 172.21.15.138:0/4066636887 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f265401e3f0 con 0x7f265c13ec70 2024-09-11T05:22:28.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:28.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26597fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:28.821 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:28.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 10 librados: watch_flush enter 2024-09-11T05:22:28.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 10 librados: watch_flush exit 2024-09-11T05:22:28.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 20 client.114162.objecter shutdown clearing up homeless session... 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 10 client.114162.objecter successfully canceled tick 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 >> v2:172.21.15.190:6800/1562782189 conn(0x7f263c03cea0 msgr2=0x7f263c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 --2- 172.21.15.138:0/4066636887 >> v2:172.21.15.190:6800/1562782189 conn(0x7f263c03cea0 0x7f263c03f360 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f265c130830 tx=0x7f264c008040 comp rx=0 tx=0).stop 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 10 monclient: shutdown 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c13ec70 msgr2=0x7f265c0708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c13ec70 0x7f265c0708d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f265400ba10 tx=0x7f265400bed0 comp rx=0 tx=0).stop 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 shutdown_connections 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 --2- 172.21.15.138:0/4066636887 >> v2:172.21.15.190:6800/1562782189 conn(0x7f263c03cea0 0x7f263c03f360 unknown :-1 s=CLOSED pgs=3486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f265c13ec70 0x7f265c0708d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:28.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f265c136c60 0x7f265c13e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:28.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 --2- 172.21.15.138:0/4066636887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f265c136170 0x7f265c1ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:28.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 >> 172.21.15.138:0/4066636887 conn(0x7f265c063640 msgr2=0x7f265c0a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:28.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 shutdown_connections 2024-09-11T05:22:28.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 -- 172.21.15.138:0/4066636887 wait complete. 2024-09-11T05:22:28.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:28.823+0000 7f26627c6640 1 librados: shutdown 2024-09-11T05:22:31.839 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:31.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient: get_monmap_and_config 2024-09-11T05:22:31.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient: build_initial_monmap 2024-09-11T05:22:31.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient: monmap: 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:31.974837+0000 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:31.974837+0000 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 1 Processor -- start 2024-09-11T05:22:31.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 1 -- start start 2024-09-11T05:22:31.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient: init 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 0x7fb14c09e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient(hunting): picked mon.noname-c con 0x7fb14c06e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb14c09e960 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb14c09ed70 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14c136510 con 0x7fb14c09ed70 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14c06e4c0 con 0x7fb14c09e960 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb14c09f820 con 0x7fb14c06e0e0 2024-09-11T05:22:31.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:31.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb15433c640 10 monclient(hunting): authenticate will time out at 1835.410948s 2024-09-11T05:22:31.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb1518d7640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7fb14c09e960 legacy=0x7fb14c1356f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:31.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb1520d8640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 msgr2=0x7fb14c09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:31.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb1528d9640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fb14c09ed70 0x7fb14c135e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50370/0 (socket says 172.21.15.138:50370) 2024-09-11T05:22:31.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb1528d9640 1 -- 172.21.15.138:0/3669396623 learned_addr learned my addr 172.21.15.138:0/3669396623 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:22:31.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.971+0000 7fb1520d8640 1 --2- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 0x7fb14c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:31.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:31.975+0000 7fb1528d9640 1 -- 172.21.15.138:0/3669396623 >> v1:172.21.15.138:6789/0 conn(0x7fb14c09ed70 legacy=0x7fb14c135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:32.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:32.175+0000 7fb1518d7640 1 -- 172.21.15.138:0/3669396623 >> v1:172.21.15.190:6789/0 conn(0x7fb14c09e960 legacy=0x7fb14c1356f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:32.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:32.175+0000 7fb1520d8640 1 -- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 msgr2=0x7fb14c09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:32.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:32.175+0000 7fb1520d8640 1 --2- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 0x7fb14c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:22:32.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:32.579+0000 7fb1520d8640 1 -- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 msgr2=0x7fb14c09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:32.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:32.579+0000 7fb1518d7640 1 -- 172.21.15.138:0/3669396623 >> v1:172.21.15.190:6789/0 conn(0x7fb14c09e960 legacy=0x7fb14c1356f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:32.576 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:32.579+0000 7fb1520d8640 1 --2- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 0x7fb14c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:22:33.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:33.383+0000 7fb1520d8640 1 -- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 msgr2=0x7fb14c09e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:33.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:33.383+0000 7fb1518d7640 1 -- 172.21.15.138:0/3669396623 >> v1:172.21.15.190:6789/0 conn(0x7fb14c09e960 legacy=0x7fb14c1356f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:33.380 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:33.383+0000 7fb1520d8640 1 --2- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 0x7fb14c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient(hunting): tick 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 monclient(hunting): continuing hunt 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 -- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 msgr2=0x7fb14c09e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 --2- 172.21.15.138:0/3669396623 >> v2:172.21.15.190:3300/0 conn(0x7fb14c06e0e0 0x7fb14c09e420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 -- 172.21.15.138:0/3669396623 >> v1:172.21.15.190:6789/0 conn(0x7fb14c09e960 legacy=0x7fb14c1356f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 -- 172.21.15.138:0/3669396623 >> v1:172.21.15.138:6789/0 conn(0x7fb14c09ed70 legacy=0x7fb14c135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:34.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient: _add_conns ranks=[3,2] 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 --2- 172.21.15.138:0/3669396623 >> v2:172.21.15.138:3300/0 conn(0x7fb138005960 0x7fb138007de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fb138005960 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 --2- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1380083d0 0x7fb13800a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fb1380083d0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 -- 172.21.15.138:0/3669396623 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fb14c09f820 con 0x7fb138005960 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 1 -- 172.21.15.138:0/3669396623 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb13800af70 con 0x7fb1380083d0 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb14bfff640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1520d8640 1 --2- 172.21.15.138:0/3669396623 >> v2:172.21.15.138:3300/0 conn(0x7fb138005960 0x7fb138007de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1520d8640 10 monclient(hunting): get_auth_request con 0x7fb138005960 auth_method 0 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1520d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1520d8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:34.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1520d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1518d7640 1 --2- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1380083d0 0x7fb13800a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1518d7640 10 monclient(hunting): get_auth_request con 0x7fb1380083d0 auth_method 0 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1518d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1518d7640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1518d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1520d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1520d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.971+0000 7fb1520d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1518d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1518d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1518d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1520d8640 10 monclient(hunting): handle_auth_done global_id 114165 payload 274 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1520d8640 1 -- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1380083d0 msgr2=0x7fb13800a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:34.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1520d8640 1 --2- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb1380083d0 0x7fb13800a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:34.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1520d8640 10 monclient: _finish_hunting 0 2024-09-11T05:22:34.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1520d8640 1 monclient: found mon.noname-a 2024-09-11T05:22:34.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1520d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:34.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1520d8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:34.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.975+0000 7fb1520d8640 1 -- 172.21.15.138:0/3669396623 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14c09f540 con 0x7fb138005960 2024-09-11T05:22:34.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1520d8640 1 -- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb138005960 msgr2=0x7fb138007de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:34.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1520d8640 1 --2- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb138005960 0x7fb138007de0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb13c003c40 tx=0x7fb13c01ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7340031d12bbc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:34.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 1 -- 172.21.15.138:0/3669396623 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0a57e0 con 0x7fb138005960 2024-09-11T05:22:34.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:34.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:34.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: dump: 2024-09-11T05:22:34.980 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:34.980 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: _finish_auth 0 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: _check_auth_tickets 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 1 -- 172.21.15.138:0/3669396623 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13c0a9070 con 0x7fb138005960 2024-09-11T05:22:34.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 1 -- 172.21.15.138:0/3669396623 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0a5be0 con 0x7fb138005960 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 5 monclient: authenticate success, global_id 114165 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 4 set_mon_vals no callback set 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1510d6640 10 monclient: dump: 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:34.982 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:34.983 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:34.983 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:34.983 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:34.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 10 monclient: shutdown 2024-09-11T05:22:34.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:34.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 -- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb138005960 msgr2=0x7fb138007de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:34.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 --2- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb138005960 0x7fb138007de0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb13c003c40 tx=0x7fb13c01ca80 comp rx=0 tx=0).stop 2024-09-11T05:22:34.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb1528d9640 1 -- 172.21.15.138:0/3669396623 reap_dead start 2024-09-11T05:22:34.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 -- 172.21.15.138:0/3669396623 shutdown_connections 2024-09-11T05:22:34.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 --2- 172.21.15.138:0/3669396623 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb138005960 0x7fb138007de0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:34.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 -- 172.21.15.138:0/3669396623 >> 172.21.15.138:0/3669396623 conn(0x7fb14c063640 msgr2=0x7fb14c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 -- 172.21.15.138:0/3669396623 shutdown_connections 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 -- 172.21.15.138:0/3669396623 wait complete. 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 10 monclient: build_initial_monmap 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 10 monclient: monmap: 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:34.987 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 librados: starting msgr at 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 librados: starting objecter 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 Processor -- start 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 -- start start 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 librados: setting wanted keys 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 1 librados: calling monclient init 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.983+0000 7fb15433c640 10 monclient: init 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb138005960 0x7fb14c131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient(hunting): picked mon.noname-c con 0x7fb138005960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb14c132500 0x7fb14c132980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:34.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient(hunting): picked mon.noname-b con 0x7fb14c132500 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb14c1d0b00 0x7fb14c1d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient(hunting): picked mon.noname-a con 0x7fb14c1d0b00 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb14c0a29c0 con 0x7fb14c1d0b00 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb14c0a1310 con 0x7fb14c132500 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb14c0a2b40 con 0x7fb138005960 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient(hunting): authenticate will time out at 1838.424334s 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb14c1d0b00 0x7fb14c1d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb14c1d0b00 0x7fb14c1d0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:38444/0 (socket says 172.21.15.138:38444) 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 1 -- 172.21.15.138:0/2827089005 learned_addr learned my addr 172.21.15.138:0/2827089005 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:34.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 10 monclient(hunting): get_auth_request con 0x7fb14c1d0b00 auth_method 0 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1518d7640 1 -- 172.21.15.138:0/2827089005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb14c132500 msgr2=0x7fb14c132980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1518d7640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb14c132500 0x7fb14c132980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb138005960 0x7fb14c131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient(hunting): get_auth_request con 0x7fb138005960 auth_method 0 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:34.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient(hunting): handle_auth_done global_id 84479 payload 824 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 -- 172.21.15.138:0/2827089005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb14c132500 msgr2=0x7fb14c132980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb14c132500 0x7fb14c132980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 -- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb14c1d0b00 msgr2=0x7fb14c1d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb14c1d0b00 0x7fb14c1d0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient: _finish_hunting 0 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 monclient: found mon.noname-c 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 -- 172.21.15.138:0/2827089005 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14c135280 con 0x7fb138005960 2024-09-11T05:22:34.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1528d9640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb14c1d0b00 0x7fb14c1d0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 -- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb138005960 msgr2=0x7fb14c131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 client.?.objecter ms_handle_connect 0x7fb138005960 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1520d8640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb138005960 0x7fb14c131fc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0ad000 tx=0x7fb13c019820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 1 -- 172.21.15.138:0/2827089005 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0aea90 con 0x7fb138005960 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: dump: 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:34.994 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: _finish_auth 0 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 5 monclient: authenticate success, global_id 84479 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 1 -- 172.21.15.138:0/2827089005 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13c0aec30 con 0x7fb138005960 2024-09-11T05:22:34.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient: _renew_subs 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb14c1337c0 con 0x7fb138005960 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 1 -- 172.21.15.138:0/2827089005 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c0b9420 con 0x7fb138005960 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb14affd640 10 monclient: dump: 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:34.996 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb1508d5640 4 set_mon_vals no callback set 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 client.84479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient: _renew_subs 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb14c134cb0 con 0x7fb138005960 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.987+0000 7fb15433c640 1 librados: init done 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb15433c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb15433c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb15433c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:34.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb14c0a2f50 con 0x7fb138005960 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb14affd640 1 -- 172.21.15.138:0/2827089005 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fb13c0c2440 con 0x7fb138005960 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb14affd640 1 --2- 172.21.15.138:0/2827089005 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb13803dd30 0x7fb1380401f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb14affd640 1 -- 172.21.15.138:0/2827089005 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(614..614 src has 520..614) ==== 5985+0+0 (secure 0 0 0) 0x7fb13c0ea2e0 con 0x7fb138005960 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb14affd640 10 client.84479.objecter ms_dispatch 0x7fb14c000ba0 osd_map(614..614 src has 520..614) 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb14affd640 3 client.84479.objecter handle_osd_map got epochs [614,614] > 0 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb14affd640 3 client.84479.objecter handle_osd_map decoding full epoch 614 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb14affd640 20 client.84479.objecter dump_active .. 0 homeless 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb1518d7640 1 --2- 172.21.15.138:0/2827089005 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb13803dd30 0x7fb1380401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.991+0000 7fb1518d7640 10 monclient: get_auth_request con 0x7fb13803dd30 auth_method 0 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.995+0000 7fb1518d7640 1 --2- 172.21.15.138:0/2827089005 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb13803dd30 0x7fb1380401f0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7fb14c13cd50 tx=0x7fb14000d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.995+0000 7fb14affd640 10 client.84479.objecter ms_handle_connect 0x7fb13803dd30 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.995+0000 7fb14affd640 1 -- 172.21.15.138:0/2827089005 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fb14c135280 con 0x7fb138005960 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.995+0000 7fb14affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:34.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:34.995+0000 7fb14affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:35.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.243+0000 7fb15433c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:35.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.243+0000 7fb15433c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:35.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.243+0000 7fb15433c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:35.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.243+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb14c09de60 con 0x7fb138005960 2024-09-11T05:22:35.241 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.243+0000 7fb14affd640 1 -- 172.21.15.138:0/2827089005 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fb13c016700 con 0x7fb138005960 2024-09-11T05:22:35.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.243+0000 7fb14affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:35.242 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.243+0000 7fb14affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:35.242 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:35.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 10 librados: watch_flush enter 2024-09-11T05:22:35.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 10 librados: watch_flush exit 2024-09-11T05:22:35.244 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 20 client.84479.objecter shutdown clearing up homeless session... 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 10 client.84479.objecter successfully canceled tick 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb13803dd30 msgr2=0x7fb1380401f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 --2- 172.21.15.138:0/2827089005 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb13803dd30 0x7fb1380401f0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7fb14c13cd50 tx=0x7fb14000d040 comp rx=0 tx=0).stop 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 10 monclient: shutdown 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb138005960 msgr2=0x7fb14c131fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb138005960 0x7fb14c131fc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0ad000 tx=0x7fb13c019820 comp rx=0 tx=0).stop 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 shutdown_connections 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 --2- 172.21.15.138:0/2827089005 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb13803dd30 0x7fb1380401f0 unknown :-1 s=CLOSED pgs=3490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb14c1d0b00 0x7fb14c1d0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb14c132500 0x7fb14c132980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 --2- 172.21.15.138:0/2827089005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb138005960 0x7fb14c131fc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:35.245 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 >> 172.21.15.138:0/2827089005 conn(0x7fb14c063640 msgr2=0x7fb14c06f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:35.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 shutdown_connections 2024-09-11T05:22:35.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 -- 172.21.15.138:0/2827089005 wait complete. 2024-09-11T05:22:35.246 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:35.247+0000 7fb15433c640 1 librados: shutdown 2024-09-11T05:22:38.262 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:22:38.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient: get_monmap_and_config 2024-09-11T05:22:38.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient: build_initial_monmap 2024-09-11T05:22:38.389 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient: monmap: 2024-09-11T05:22:38.389 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:38.389 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:38.395254+0000 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:38.395254+0000 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:38.390 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 1 Processor -- start 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 1 -- start start 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient: init 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40680a5b40 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient(hunting): picked mon.noname-b con 0x7f40680a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient(hunting): picked mon.noname-a con 0x7f406809d840 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:38.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406809d490 con 0x7f40680a5b40 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f406809e110 con 0x7f406809d840 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4068070580 con 0x7f40680a7130 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f406d771640 10 monclient(hunting): authenticate will time out at 1841.831459s 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f4066ffd640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7f40680a5b40 legacy=0x7f40680a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59044/0 (socket says 172.21.15.138:59044) 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 learned_addr learned my addr 172.21.15.138:0/2967514260 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 10 monclient(hunting): get_auth_request con 0x7f40680a7130 auth_method 0 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:38.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40677fe640 10 monclient(hunting): get_auth_request con 0x7f406809d840 auth_method 0 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.391+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:38.395 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.395+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:38.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f4066ffd640 1 -- 172.21.15.138:0/2967514260 >> v1:172.21.15.190:6789/0 conn(0x7f40680a5b40 legacy=0x7f40680a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:38.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:38.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 10 monclient(hunting): get_auth_request con 0x7f40680a7130 auth_method 0 2024-09-11T05:22:38.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:38.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:38.592 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:38.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:22:38.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:38.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:38.593 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.595+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:22:38.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:38.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 10 monclient(hunting): get_auth_request con 0x7f406809d840 auth_method 0 2024-09-11T05:22:38.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:38.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:38.596 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:38.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:22:38.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:38.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:38.597 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.599+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:22:38.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f4066ffd640 1 -- 172.21.15.138:0/2967514260 >> v1:172.21.15.190:6789/0 conn(0x7f40680a5b40 legacy=0x7f40680a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:38.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:38.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 10 monclient(hunting): get_auth_request con 0x7f40680a7130 auth_method 0 2024-09-11T05:22:38.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:38.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:38.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:38.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:22:38.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:38.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:38.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:38.999+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 10 monclient(hunting): get_auth_request con 0x7f406809d840 auth_method 0 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:39.001 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.003+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f4066ffd640 1 -- 172.21.15.138:0/2967514260 >> v1:172.21.15.190:6789/0 conn(0x7f40680a5b40 legacy=0x7f40680a5f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 10 monclient(hunting): get_auth_request con 0x7f40680a7130 auth_method 0 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:39.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.803+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 10 monclient(hunting): get_auth_request con 0x7f406809d840 auth_method 0 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:22:39.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:39.807+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:22:41.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 10 monclient(hunting): tick 2024-09-11T05:22:41.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:22:41.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 monclient(hunting): continuing hunt 2024-09-11T05:22:41.391 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 msgr2=0x7f406813db40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f40680a7130 0x7f406813db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 msgr2=0x7f4068140090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.138:3300/0 conn(0x7f406809d840 0x7f4068140090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 -- 172.21.15.138:0/2967514260 >> v1:172.21.15.190:6789/0 conn(0x7f40680a5b40 legacy=0x7f40680a5f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 10 monclient: _add_conns ranks=[0,4] 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40540036d0 addr v1:172.21.15.138:6789/0 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.190:3300/0 conn(0x7f40540073e0 0x7f40540098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f40540073e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 -- 172.21.15.138:0/2967514260 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406809cb80 con 0x7f40540036d0 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 1 -- 172.21.15.138:0/2967514260 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f406809e110 con 0x7f40540073e0 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.391+0000 7f4064ff9640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:41.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.395+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.190:3300/0 conn(0x7f40540073e0 msgr2=0x7f40540098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:41.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.395+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.190:3300/0 conn(0x7f40540073e0 0x7f40540098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:41.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.395+0000 7f4066ffd640 1 -- 172.21.15.138:0/2967514260 >> v1:172.21.15.138:6789/0 conn(0x7f40540036d0 legacy=0x7f4054005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.394 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.395+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:22:41.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100146658 0 0) 0x7f406809cb80 con 0x7f40540036d0 2024-09-11T05:22:41.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): _negotiate 2024-09-11T05:22:41.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): my global_id is 114171 2024-09-11T05:22:41.396 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4050003530 con 0x7f40540036d0 2024-09-11T05:22:41.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 830197962 0 0) 0x7f4050003530 con 0x7f40540036d0 2024-09-11T05:22:41.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 >> v2:172.21.15.190:3300/0 conn(0x7f40540073e0 msgr2=0x7f40540098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:41.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 --2- 172.21.15.138:0/2967514260 >> v2:172.21.15.190:3300/0 conn(0x7f40540073e0 0x7f40540098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: _finish_hunting 0 2024-09-11T05:22:41.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:22:41.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:41.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406809d490 con 0x7f40540036d0 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: _finish_auth 0 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f405c004650 con 0x7f40540036d0 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f406d771640 5 monclient: authenticate success, global_id 114171 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f406d771640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: dump: 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.399 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 >> v1:172.21.15.138:6789/0 conn(0x7f40540036d0 legacy=0x7f4054005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 1 -- 172.21.15.138:0/2967514260 reap_dead start 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:22:41.400 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40500039c0 0x7f4050003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): picked mon.a con 0x7f40500039c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4050004810 0x7f4050004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): picked mon.b con 0x7f4050004810 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f405000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): picked mon.c con 0x7f4050005380 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f406809e110 con 0x7f40500039c0 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f405000c2d0 con 0x7f4050004810 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f405000c4d0 con 0x7f4050005380 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4065ffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:22:41.401 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f406d771640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40500039c0 0x7f4050003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40667fc640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4050004810 msgr2=0x7f4050004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40667fc640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4050004810 0x7f4050004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient(hunting): get_auth_request con 0x7f40500039c0 auth_method 0 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f405000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 10 monclient(hunting): get_auth_request con 0x7f4050005380 auth_method 0 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.402 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f40677fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient(hunting): handle_auth_done global_id 114171 payload 293 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 msgr2=0x7f405000b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f405000b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4050004810 msgr2=0x7f4050004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4050004810 0x7f4050004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 1 monclient: found mon.a 2024-09-11T05:22:41.403 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:41.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 1 -- 172.21.15.138:0/2967514260 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4050003530 con 0x7f40500039c0 2024-09-11T05:22:41.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient: _finish_auth 0 2024-09-11T05:22:41.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:41.404 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.399+0000 7f4066ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:41.405 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4066ffd640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40500039c0 msgr2=0x7f4050003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.406 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4066ffd640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40500039c0 0x7f4050003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f405c003940 tx=0x7f405c01cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb65978127f46e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c00c200 con 0x7f40500039c0 2024-09-11T05:22:41.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:41.408 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 10 monclient: dump: 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c0a9070 con 0x7f40500039c0 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f406d771640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:41.409 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f406d771640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f406d771640 4 set_mon_vals no callback set 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 1 -- 172.21.15.138:0/2967514260 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c004920 con 0x7f40500039c0 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f4065ffb640 10 monclient: dump: 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.410 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f406d771640 10 monclient: shutdown 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f406d771640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f406d771640 1 -- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40500039c0 msgr2=0x7f4050003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.407+0000 7f406d771640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40500039c0 0x7f4050003dc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f405c003940 tx=0x7f405c01cea0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 -- 172.21.15.138:0/2967514260 shutdown_connections 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f405000b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4050004810 0x7f4050004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 --2- 172.21.15.138:0/2967514260 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f40500039c0 0x7f4050003dc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 -- 172.21.15.138:0/2967514260 >> 172.21.15.138:0/2967514260 conn(0x7f40680aba10 msgr2=0x7f40680a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 -- 172.21.15.138:0/2967514260 shutdown_connections 2024-09-11T05:22:41.411 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 -- 172.21.15.138:0/2967514260 wait complete. 2024-09-11T05:22:41.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient: build_initial_monmap 2024-09-11T05:22:41.415 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient: monmap: 2024-09-11T05:22:41.415 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:41.415 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 librados: starting msgr at 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 librados: starting objecter 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 Processor -- start 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 -- start start 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 librados: setting wanted keys 2024-09-11T05:22:41.416 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 librados: calling monclient init 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient: init 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40500039c0 0x7f4068137a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient(hunting): picked mon.noname-b con 0x7f40500039c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4050004810 0x7f4068137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient(hunting): picked mon.noname-a con 0x7f4050004810 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f4068125620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient(hunting): picked mon.noname-c con 0x7f4050005380 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4068138900 con 0x7f4050004810 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f40680a6940 con 0x7f40500039c0 2024-09-11T05:22:41.417 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4068138a80 con 0x7f4050005380 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f406d771640 10 monclient(hunting): authenticate will time out at 1844.851691s 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f4066ffd640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40500039c0 msgr2=0x7f4068137a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f4066ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40500039c0 0x7f4068137a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40667fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4050004810 0x7f4068137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40677fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f4068125620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40667fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4050004810 0x7f4068137f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43068/0 (socket says 172.21.15.138:43068) 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40667fc640 1 -- 172.21.15.138:0/2649988510 learned_addr learned my addr 172.21.15.138:0/2649988510 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40667fc640 10 monclient(hunting): get_auth_request con 0x7f4050004810 auth_method 0 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40667fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.418 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40677fe640 10 monclient(hunting): get_auth_request con 0x7f4050005380 auth_method 0 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.411+0000 7f40677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 10 monclient(hunting): handle_auth_done global_id 114174 payload 824 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 1 -- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 msgr2=0x7f4068125620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 1 --2- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f4068125620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 1 -- 172.21.15.138:0/2649988510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40500039c0 msgr2=0x7f4068137a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 1 --2- 172.21.15.138:0/2649988510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40500039c0 0x7f4068137a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.419 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 10 monclient: _finish_hunting 0 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 1 monclient: found mon.noname-a 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 1 -- 172.21.15.138:0/2649988510 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4068129160 con 0x7f4050004810 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40677fe640 1 --2- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f4068125620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 1 -- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4050004810 msgr2=0x7f4068137f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40667fc640 1 --2- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4050004810 0x7f4068137f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4058014670 tx=0x7f4058014b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 client.?.objecter ms_handle_connect 0x7f4050004810 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 1 -- 172.21.15.138:0/2649988510 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405801d070 con 0x7f4050004810 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: dump: 2024-09-11T05:22:41.420 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 1 -- 172.21.15.138:0/2649988510 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4058012c60 con 0x7f4050004810 2024-09-11T05:22:41.421 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 1 -- 172.21.15.138:0/2649988510 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405800f040 con 0x7f4050004810 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 5 monclient: authenticate success, global_id 114174 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 10 monclient: _renew_subs 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f40657fa640 4 set_mon_vals no callback set 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4068129440 con 0x7f4050004810 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f404f7fe640 10 monclient: dump: 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.422 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 10 client.114174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 10 monclient: _renew_subs 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4068127ff0 con 0x7f4050004810 2024-09-11T05:22:41.423 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.415+0000 7f406d771640 1 librados: init done 2024-09-11T05:22:41.424 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f406d771640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f406d771640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f406d771640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f404f7fe640 1 -- 172.21.15.138:0/2649988510 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f4058015040 con 0x7f4050004810 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4068138d80 con 0x7f4050004810 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f404f7fe640 1 --2- 172.21.15.138:0/2649988510 >> v2:172.21.15.190:6800/1562782189 conn(0x7f405403d690 0x7f405403fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f404f7fe640 1 -- 172.21.15.138:0/2649988510 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(615..615 src has 520..615) ==== 5985+0+0 (secure 0 0 0) 0x7f40580502b0 con 0x7f4050004810 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f404f7fe640 10 client.114174.objecter ms_dispatch 0x7f4068000ba0 osd_map(615..615 src has 520..615) 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f404f7fe640 3 client.114174.objecter handle_osd_map got epochs [615,615] > 0 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f404f7fe640 3 client.114174.objecter handle_osd_map decoding full epoch 615 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f404f7fe640 20 client.114174.objecter dump_active .. 0 homeless 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f4066ffd640 1 --2- 172.21.15.138:0/2649988510 >> v2:172.21.15.190:6800/1562782189 conn(0x7f405403d690 0x7f405403fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.425 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.419+0000 7f4066ffd640 10 monclient: get_auth_request con 0x7f405403d690 auth_method 0 2024-09-11T05:22:41.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.423+0000 7f4066ffd640 1 --2- 172.21.15.138:0/2649988510 >> v2:172.21.15.190:6800/1562782189 conn(0x7f405403d690 0x7f405403fb50 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7f405c009000 tx=0x7f405c0a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.423+0000 7f404f7fe640 10 client.114174.objecter ms_handle_connect 0x7f405403d690 2024-09-11T05:22:41.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.423+0000 7f404f7fe640 1 -- 172.21.15.138:0/2649988510 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f4058012e10 con 0x7f4050004810 2024-09-11T05:22:41.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.423+0000 7f404f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:41.426 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.423+0000 7f404f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:41.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:22:41.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:41.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f40681270e0 con 0x7f4050004810 2024-09-11T05:22:41.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f404f7fe640 1 -- 172.21.15.138:0/2649988510 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f405804e7f0 con 0x7f4050004810 2024-09-11T05:22:41.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f404f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:22:41.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f404f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:41.658 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:22:41.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 10 librados: watch_flush enter 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 10 librados: watch_flush exit 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 20 client.114174.objecter shutdown clearing up homeless session... 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 10 client.114174.objecter successfully canceled tick 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 >> v2:172.21.15.190:6800/1562782189 conn(0x7f405403d690 msgr2=0x7f405403fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 --2- 172.21.15.138:0/2649988510 >> v2:172.21.15.190:6800/1562782189 conn(0x7f405403d690 0x7f405403fb50 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7f405c009000 tx=0x7f405c0a7070 comp rx=0 tx=0).stop 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 10 monclient: shutdown 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4050004810 msgr2=0x7f4068137f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 --2- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4050004810 0x7f4068137f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4058014670 tx=0x7f4058014b30 comp rx=0 tx=0).stop 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 shutdown_connections 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 --2- 172.21.15.138:0/2649988510 >> v2:172.21.15.190:6800/1562782189 conn(0x7f405403d690 0x7f405403fb50 unknown :-1 s=CLOSED pgs=3491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 --2- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4050005380 0x7f4068125620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 --2- 172.21.15.138:0/2649988510 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4050004810 0x7f4068137f80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 --2- 172.21.15.138:0/2649988510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40500039c0 0x7f4068137a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 >> 172.21.15.138:0/2649988510 conn(0x7f40680aba10 msgr2=0x7f40680a3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:41.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.659+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 shutdown_connections 2024-09-11T05:22:41.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.663+0000 7f406d771640 1 -- 172.21.15.138:0/2649988510 wait complete. 2024-09-11T05:22:41.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.663+0000 7f406d771640 1 librados: shutdown 2024-09-11T05:22:41.675 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:22:41.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T05:22:41.675 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:22:41.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient: get_monmap_and_config 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient: build_initial_monmap 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient: monmap: 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:41.802558+0000 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:41.802558+0000 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:41.797 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:41.798 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 1 Processor -- start 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 1 -- start start 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient: init 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5100a32c0 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51013a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient(hunting): picked mon.noname-b con 0x7ff51009eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:41.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff51009ef50 0x7ff51013ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient(hunting): picked mon.noname-a con 0x7ff51009ef50 addr v2:172.21.15.138:3300/0 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5100a1b50 con 0x7ff5100a32c0 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7ff5100a1920 con 0x7ff51009ef50 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff5100a36a0 con 0x7ff51009eb30 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff515cd9640 10 monclient(hunting): authenticate will time out at 1845.239158s 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51013a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7ff51009ef50 0x7ff51013ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51013a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59096/0 (socket says 172.21.15.138:59096) 2024-09-11T05:22:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 1 -- 172.21.15.138:0/3514013005 learned_addr learned my addr 172.21.15.138:0/3514013005 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ff5100a32c0 legacy=0x7ff51009e420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient(hunting): get_auth_request con 0x7ff51009eb30 auth_method 0 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 10 monclient(hunting): get_auth_request con 0x7ff51009ef50 auth_method 0 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient(hunting): handle_auth_done global_id 84485 payload 274 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 1 -- 172.21.15.138:0/3514013005 >> v2:172.21.15.138:3300/0 conn(0x7ff51009ef50 msgr2=0x7ff51013ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 1 --2- 172.21.15.138:0/3514013005 >> v2:172.21.15.138:3300/0 conn(0x7ff51009ef50 0x7ff51013ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 1 -- 172.21.15.138:0/3514013005 >> v1:172.21.15.190:6789/0 conn(0x7ff5100a32c0 legacy=0x7ff51009e420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 1 monclient: found mon.noname-b 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff506ffd640 1 -- 172.21.15.138:0/3514013005 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51009f560 con 0x7ff51009eb30 2024-09-11T05:22:41.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.799+0000 7ff50ffff640 1 --2- 172.21.15.138:0/3514013005 >> v2:172.21.15.138:3300/0 conn(0x7ff51009ef50 0x7ff51013ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.803+0000 7ff506ffd640 1 -- 172.21.15.138:0/3514013005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 msgr2=0x7ff51013a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.803+0000 7ff506ffd640 1 --2- 172.21.15.138:0/3514013005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51013a930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009870 tx=0x7ff4f801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16989d8820ba7a24 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 1 -- 172.21.15.138:0/3514013005 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80b0070 con 0x7ff51009eb30 2024-09-11T05:22:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: dump: 2024-09-11T05:22:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: _finish_auth 0 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 5 monclient: authenticate success, global_id 84485 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 1 -- 172.21.15.138:0/3514013005 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f80ab600 con 0x7ff51009eb30 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 1 -- 172.21.15.138:0/3514013005 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80ab9d0 con 0x7ff51009eb30 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50effd640 10 monclient: dump: 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 4 set_mon_vals no callback set 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient: shutdown 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- 172.21.15.138:0/3514013005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 msgr2=0x7ff51013a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 --2- 172.21.15.138:0/3514013005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51013a930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009870 tx=0x7ff4f801c3f0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- 172.21.15.138:0/3514013005 shutdown_connections 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 --2- 172.21.15.138:0/3514013005 >> v2:172.21.15.138:3300/0 conn(0x7ff51009ef50 0x7ff51013ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 --2- 172.21.15.138:0/3514013005 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51013a930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- 172.21.15.138:0/3514013005 >> 172.21.15.138:0/3514013005 conn(0x7ff510063640 msgr2=0x7ff510063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- 172.21.15.138:0/3514013005 shutdown_connections 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- 172.21.15.138:0/3514013005 wait complete. 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient: build_initial_monmap 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient: monmap: 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 librados: starting msgr at 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 librados: starting objecter 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 Processor -- start 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- start start 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 librados: setting wanted keys 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 librados: calling monclient init 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient: init 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51011d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient(hunting): picked mon.noname-c con 0x7ff51009eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51009ef50 0x7ff51011dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient(hunting): picked mon.noname-a con 0x7ff51009ef50 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5100a32c0 0x7ff51011dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient(hunting): picked mon.noname-b con 0x7ff5100a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:41.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff5100a12d0 con 0x7ff51009ef50 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff5100a29c0 con 0x7ff5100a32c0 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7ff51011fe30 con 0x7ff51009eb30 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff515cd9640 10 monclient(hunting): authenticate will time out at 1845.246216s 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51011d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51009ef50 0x7ff51011dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51009ef50 0x7ff51011dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43088/0 (socket says 172.21.15.138:43088) 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 -- 172.21.15.138:0/1660469239 learned_addr learned my addr 172.21.15.138:0/1660469239 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51011d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59104/0 (socket says 172.21.15.138:59104) 2024-09-11T05:22:41.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient(hunting): get_auth_request con 0x7ff51009ef50 auth_method 0 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 10 monclient(hunting): get_auth_request con 0x7ff51009eb30 auth_method 0 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50ffff640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5100a32c0 msgr2=0x7ff51011dff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50ffff640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5100a32c0 0x7ff51011dff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient(hunting): handle_auth_done global_id 114180 payload 824 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 msgr2=0x7ff51011d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51011d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5100a32c0 msgr2=0x7ff51011dff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5100a32c0 0x7ff51011dff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 monclient: found mon.noname-a 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff506ffd640 1 -- 172.21.15.138:0/1660469239 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51011f870 con 0x7ff51009ef50 2024-09-11T05:22:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.807+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7ff51009eb30 0x7ff51011d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff506ffd640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51009ef50 msgr2=0x7ff51011dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff506ffd640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51009ef50 0x7ff51011dab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80ae040 tx=0x7ff4f801ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 client.?.objecter ms_handle_connect 0x7ff51009ef50 2024-09-11T05:22:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80ba020 con 0x7ff51009ef50 2024-09-11T05:22:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: dump: 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: _finish_auth 0 2024-09-11T05:22:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: _check_auth_tickets 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f80b6630 con 0x7ff51009ef50 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 5 monclient: authenticate success, global_id 114180 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 10 monclient: _renew_subs 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff51011fb50 con 0x7ff51009ef50 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80b0040 con 0x7ff51009ef50 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 monclient: dump: 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50e7fc640 4 set_mon_vals no callback set 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 10 client.114180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 10 monclient: _renew_subs 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51009e110 con 0x7ff51009ef50 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 1 librados: init done 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51013b7b0 0x7ff510135380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff5100a1450 con 0x7ff51013b7b0 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7ff4f80b3030 con 0x7ff51009ef50 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51013b7b0 0x7ff510135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 10 monclient: get_auth_request con 0x7ff51013b7b0 auth_method 0 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_done global_id 114183 payload 824 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 1 --2- 172.21.15.138:0/1660469239 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4f003ce50 0x7ff4f003f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1660469239 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5100a2da0 con 0x7ff51013b7b0 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(616..616 src has 520..616) ==== 6014+0+0 (secure 0 0 0) 0x7ff4f80e9b00 con 0x7ff51009ef50 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 10 client.114180.objecter ms_dispatch 0x7ff510000ba0 osd_map(616..616 src has 520..616) 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 3 client.114180.objecter handle_osd_map got epochs [616,616] > 0 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 3 client.114180.objecter handle_osd_map decoding full epoch 616 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50cff9640 20 client.114180.objecter dump_active .. 0 homeless 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50ffff640 1 --2- 172.21.15.138:0/1660469239 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4f003ce50 0x7ff4f003f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.811+0000 7ff50ffff640 10 monclient: get_auth_request con 0x7ff4f003ce50 auth_method 0 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 10 client.114180.objecter ms_handle_connect 0x7ff51013b7b0 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51013b7b0 msgr2=0x7ff510135380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 10 client.114180.objecter resend_mon_ops 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51013b7b0 0x7ff510135380 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5000052f0 tx=0x7ff5000057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500016070 con 0x7ff51013b7b0 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 10 monclient: dump: 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff50000ed60 con 0x7ff51013b7b0 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:41.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51013b7b0 msgr2=0x7ff510135380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51013b7b0 0x7ff510135380 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5000052f0 tx=0x7ff5000057b0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51013b7b0 msgr2=0x7ff510135380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51013b7b0 0x7ff510135380 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5000052f0 tx=0x7ff5000057b0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50cff9640 10 client.114180.objecter ms_handle_connect 0x7ff4f003ce50 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.815+0000 7ff50ffff640 1 --2- 172.21.15.138:0/1660469239 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4f003ce50 0x7ff4f003f310 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7ff50800bde0 tx=0x7ff50800d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff515cd9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff515cd9640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51002d450 0x7ff51002d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff515cd9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7ff510138020 con 0x7ff51002d450 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51002d450 0x7ff51002d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 10 monclient: get_auth_request con 0x7ff51002d450 auth_method 0 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 10 monclient(hunting): handle_auth_done global_id 114186 payload 824 2024-09-11T05:22:41.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.819+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1660469239 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff51006cca0 con 0x7ff51002d450 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50f7fe640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51002d450 msgr2=0x7ff51002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 10 client.114180.objecter ms_handle_connect 0x7ff51002d450 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 10 client.114180.objecter resend_mon_ops 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50f7fe640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51002d450 0x7ff51002d860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff500018000 tx=0x7ff500005990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500016040 con 0x7ff51002d450 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 10 monclient: dump: 2024-09-11T05:22:41.828 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.829 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7ff500005c10 con 0x7ff51002d450 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51002d450 msgr2=0x7ff51002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51002d450 0x7ff51002d860 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff500018000 tx=0x7ff500005990 comp rx=0 tx=0).stop 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51002d450 msgr2=0x7ff51002d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50cff9640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51002d450 0x7ff51002d860 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff500018000 tx=0x7ff500005990 comp rx=0 tx=0).stop 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:41.830 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 62, 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 33863, 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:41.831 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:41.832 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:22:41.833 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:41.834 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.835 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:22:41.836 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:41.837 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:22:41.838 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:41.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:41.841 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:22:41.842 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:22:41.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 10 librados: watch_flush enter 2024-09-11T05:22:41.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 10 librados: watch_flush exit 2024-09-11T05:22:41.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 20 client.114180.objecter shutdown clearing up homeless session... 2024-09-11T05:22:41.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 10 client.114180.objecter successfully canceled tick 2024-09-11T05:22:41.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4f003ce50 msgr2=0x7ff4f003f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 --2- 172.21.15.138:0/1660469239 >> v2:172.21.15.190:6800/1562782189 conn(0x7ff4f003ce50 0x7ff4f003f310 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7ff50800bde0 tx=0x7ff50800d040 comp rx=0 tx=0).stop 2024-09-11T05:22:41.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 10 monclient: shutdown 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51009ef50 msgr2=0x7ff51011dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 --2- 172.21.15.138:0/1660469239 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7ff51009ef50 0x7ff51011dab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80ae040 tx=0x7ff4f801ac30 comp rx=0 tx=0).stop 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50ffff640 1 -- 172.21.15.138:0/1660469239 reap_dead start 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff50ffff640 1 -- 172.21.15.138:0/1660469239 reap_dead start 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 shutdown_connections 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 >> 172.21.15.138:0/1660469239 conn(0x7ff510063640 msgr2=0x7ff51006f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 shutdown_connections 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 -- 172.21.15.138:0/1660469239 wait complete. 2024-09-11T05:22:41.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.823+0000 7ff515cd9640 1 librados: shutdown 2024-09-11T05:22:41.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:22:41.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T05:22:41.846 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:22:41.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient: get_monmap_and_config 2024-09-11T05:22:41.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient: build_initial_monmap 2024-09-11T05:22:41.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient: monmap: 2024-09-11T05:22:41.952 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:41.952 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.952 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:22:41.957908+0000 2024-09-11T05:22:41.952 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:22:41.957908+0000 2024-09-11T05:22:41.952 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:41.953 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.953 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:22:41.953 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:22:41.953 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:22:41.953 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:22:41.953 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:22:41.953 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.954 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 1 Processor -- start 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 1 -- start start 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient: init 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c84070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8409e6e0 0x7f5c8409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient(hunting): picked mon.noname-b con 0x7f5c8409e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c8413d640 0x7f5c8413fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient(hunting): picked mon.noname-c con 0x7f5c8413d640 addr v2:172.21.15.190:3300/0 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c84135b60 con 0x7f5c84070010 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5c840703f0 con 0x7f5c8413d640 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.955 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5c840a8990 con 0x7f5c8409e6e0 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c88d0d640 10 monclient(hunting): authenticate will time out at 1845.393828s 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c8259c640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7f5c84070010 legacy=0x7f5c84135450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8409e6e0 0x7f5c8409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8409e6e0 0x7f5c8409eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:59106/0 (socket says 172.21.15.138:59106) 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 1 -- 172.21.15.138:0/85794715 learned_addr learned my addr 172.21.15.138:0/85794715 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient(hunting): get_auth_request con 0x7f5c8409e6e0 auth_method 0 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c82d9d640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7f5c8413d640 msgr2=0x7f5c8413fa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c82d9d640 1 --2- 172.21.15.138:0/85794715 >> v2:172.21.15.190:3300/0 conn(0x7f5c8413d640 0x7f5c8413fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient(hunting): handle_auth_done global_id 84491 payload 274 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 1 -- 172.21.15.138:0/85794715 >> v2:172.21.15.190:3300/0 conn(0x7f5c8413d640 msgr2=0x7f5c8413fa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 1 --2- 172.21.15.138:0/85794715 >> v2:172.21.15.190:3300/0 conn(0x7f5c8413d640 0x7f5c8413fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 1 -- 172.21.15.138:0/85794715 >> v1:172.21.15.190:6789/0 conn(0x7f5c84070010 legacy=0x7f5c84135450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 1 monclient: found mon.noname-b 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:22:41.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.955+0000 7f5c81d9b640 1 -- 172.21.15.138:0/85794715 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8409cbf0 con 0x7f5c8409e6e0 2024-09-11T05:22:41.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c81d9b640 1 -- 172.21.15.138:0/85794715 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8409e6e0 msgr2=0x7f5c8409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c81d9b640 1 --2- 172.21.15.138:0/85794715 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8409e6e0 0x7f5c8409eae0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009870 tx=0x7f5c7801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96f046d295b496b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 1 -- 172.21.15.138:0/85794715 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c780b0070 con 0x7f5c8409e6e0 2024-09-11T05:22:41.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:22:41.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: dump: 2024-09-11T05:22:41.961 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.961 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: _finish_auth 0 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: _check_auth_tickets 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 5 monclient: authenticate success, global_id 84491 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:22:41.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 1 -- 172.21.15.138:0/85794715 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c780ab600 con 0x7f5c8409e6e0 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 10 monclient: get_monmap_and_config success 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 4 set_mon_vals no callback set 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 1 -- 172.21.15.138:0/85794715 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c780ab9d0 con 0x7f5c8409e6e0 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c8159a640 10 monclient: dump: 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.963 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 10 monclient: shutdown 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 -- 172.21.15.138:0/85794715 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8409e6e0 msgr2=0x7f5c8409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 --2- 172.21.15.138:0/85794715 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8409e6e0 0x7f5c8409eae0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009870 tx=0x7f5c7801c440 comp rx=0 tx=0).stop 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 -- 172.21.15.138:0/85794715 shutdown_connections 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 --2- 172.21.15.138:0/85794715 >> v2:172.21.15.190:3300/0 conn(0x7f5c8413d640 0x7f5c8413fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 --2- 172.21.15.138:0/85794715 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8409e6e0 0x7f5c8409eae0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 -- 172.21.15.138:0/85794715 >> 172.21.15.138:0/85794715 conn(0x7f5c840ab830 msgr2=0x7f5c840a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 -- 172.21.15.138:0/85794715 shutdown_connections 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 -- 172.21.15.138:0/85794715 wait complete. 2024-09-11T05:22:41.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 10 monclient: build_initial_monmap 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 10 monclient: monmap: 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 librados: starting msgr at 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.959+0000 7f5c88d0d640 1 librados: starting objecter 2024-09-11T05:22:41.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 Processor -- start 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 -- start start 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 librados: setting wanted keys 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 librados: calling monclient init 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient: init 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient: _reopen_session rank -1 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c84070010 0x7f5c841d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient(hunting): picked mon.noname-b con 0x7f5c84070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5c8409e6e0 0x7f5c8413c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient(hunting): picked mon.noname-a con 0x7f5c8409e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8413d640 0x7f5c8413ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient(hunting): picked mon.noname-c con 0x7f5c8413d640 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5c84142bd0 con 0x7f5c8409e6e0 2024-09-11T05:22:41.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5c84142a50 con 0x7f5c84070010 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5c84142d50 con 0x7f5c8413d640 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient(hunting): _renew_subs 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient(hunting): authenticate will time out at 1845.400585s 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5c8409e6e0 0x7f5c8413c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8413d640 0x7f5c8413ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5c8409e6e0 0x7f5c8413c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43106/0 (socket says 172.21.15.138:43106) 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c8259c640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c84070010 msgr2=0x7f5c841d0350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 1 -- 172.21.15.138:0/1485482184 learned_addr learned my addr 172.21.15.138:0/1485482184 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 10 monclient(hunting): get_auth_request con 0x7f5c8409e6e0 auth_method 0 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient(hunting): get_auth_request con 0x7f5c8413d640 auth_method 0 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c8259c640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c84070010 0x7f5c841d0350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient(hunting): handle_auth_done global_id 84494 payload 824 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c84070010 msgr2=0x7f5c841d0350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:22:41.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c84070010 0x7f5c841d0350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5c8409e6e0 msgr2=0x7f5c8413c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5c8409e6e0 0x7f5c8413c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:22:41.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 monclient: found mon.noname-c 2024-09-11T05:22:41.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:22:41.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:22:41.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 -- 172.21.15.138:0/1485482184 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8413fcb0 con 0x7f5c8413d640 2024-09-11T05:22:41.969 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c81d9b640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5c8409e6e0 0x7f5c8413c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:22:41.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8413d640 msgr2=0x7f5c8413ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8413d640 0x7f5c8413ca30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00bd90 tx=0x7f5c7c009eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 10 client.?.objecter ms_handle_connect 0x7f5c8413d640 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c82d9d640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8413d640 0x7f5c8413ca30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00bd90 tx=0x7f5c7c009eb0 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f5c7c0066f0 mon_map magic: 0 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c0066f0 con 0x7f5c8413d640 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 10 monclient: dump: 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.974 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 10 monclient: _finish_auth 0 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c7c006890 con 0x7f5c8413d640 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 5 monclient: authenticate success, global_id 84494 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient: _renew_subs 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c84137cb0 con 0x7f5c8413d640 2024-09-11T05:22:41.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c72ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c80d99640 4 set_mon_vals no callback set 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 client.84494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient: _renew_subs 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c84136790 con 0x7f5c8413d640 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.963+0000 7f5c88d0d640 1 librados: init done 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c88d0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c88d0d640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c841376f0 0x7f5c840a3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c88d0d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8409eae0 con 0x7f5c841376f0 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c841376f0 0x7f5c840a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 10 monclient: get_auth_request con 0x7f5c841376f0 auth_method 0 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 10 monclient(hunting): handle_auth_done global_id 84497 payload 824 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c015440 con 0x7f5c8413d640 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 1 -- 172.21.15.138:0/1485482184 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c840a53e0 con 0x7f5c841376f0 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 monclient: dump: 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.977 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5c7c01d030 con 0x7f5c8413d640 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 --2- 172.21.15.138:0/1485482184 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5c6403ce50 0x7f5c6403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c81d9b640 1 --2- 172.21.15.138:0/1485482184 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5c6403ce50 0x7f5c6403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c81d9b640 10 monclient: get_auth_request con 0x7f5c6403ce50 auth_method 0 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 client.84494.objecter ms_handle_connect 0x7f5c841376f0 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 client.84494.objecter resend_mon_ops 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c841376f0 msgr2=0x7f5c840a3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c8259c640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c841376f0 0x7f5c840a3920 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400b450 tx=0x7f5c7400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74005270 con 0x7f5c841376f0 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 monclient: dump: 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c74015070 con 0x7f5c841376f0 2024-09-11T05:22:41.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c841376f0 msgr2=0x7f5c840a3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c841376f0 0x7f5c840a3920 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400b450 tx=0x7f5c7400b910 comp rx=0 tx=0).stop 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c841376f0 msgr2=0x7f5c840a3920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c72ffd640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c841376f0 0x7f5c840a3920 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.971+0000 7f5c72ffd640 10 client.84494.objecter ms_handle_connect 0x7f5c6403ce50 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.967+0000 7f5c81d9b640 1 --2- 172.21.15.138:0/1485482184 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5c6403ce50 0x7f5c6403f310 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f5c7801c270 tx=0x7f5c780a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c88d0d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c88d0d640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c840a3e60 0x7f5c840a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c88d0d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5c840a47b0 con 0x7f5c840a3e60 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c840a3e60 0x7f5c840a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:22:41.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 10 monclient: get_auth_request con 0x7f5c840a3e60 auth_method 0 2024-09-11T05:22:41.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:22:41.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:22:41.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:22:41.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:22:41.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:22:41.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:22:41.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 10 monclient(hunting): handle_auth_done global_id 84500 payload 824 2024-09-11T05:22:41.981 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 1 -- 172.21.15.138:0/1485482184 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c84069fa0 con 0x7f5c840a3e60 2024-09-11T05:22:41.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c840a3e60 msgr2=0x7f5c840a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:22:41.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c72ffd640 10 client.84494.objecter ms_handle_connect 0x7f5c840a3e60 2024-09-11T05:22:41.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c72ffd640 10 client.84494.objecter resend_mon_ops 2024-09-11T05:22:41.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.975+0000 7f5c8259c640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c840a3e60 0x7f5c840a4270 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400b450 tx=0x7f5c74005a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:22:41.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74010690 con 0x7f5c840a3e60 2024-09-11T05:22:41.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:22:41.982 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 10 monclient: dump: 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f5c74010690 con 0x7f5c840a3e60 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:22:41.983 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c840a3e60 msgr2=0x7f5c840a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c840a3e60 0x7f5c840a4270 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400b450 tx=0x7f5c74005a30 comp rx=0 tx=0).stop 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c840a3e60 msgr2=0x7f5c840a4270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c840a3e60 0x7f5c840a4270 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400b450 tx=0x7f5c74005a30 comp rx=0 tx=0).stop 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 62, 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 405749, 2024-09-11T05:22:41.984 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:41.985 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:41.986 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:41.987 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:22:41.988 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:22:41.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:41.990 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:22:41.991 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:22:41.993 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:22:41.994 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.995 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:22:41.996 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:22:41.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 10 librados: watch_flush enter 2024-09-11T05:22:41.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 10 librados: watch_flush exit 2024-09-11T05:22:41.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 20 client.84494.objecter shutdown clearing up homeless session... 2024-09-11T05:22:41.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 10 client.84494.objecter successfully canceled tick 2024-09-11T05:22:41.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5c6403ce50 msgr2=0x7f5c6403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 --2- 172.21.15.138:0/1485482184 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5c6403ce50 0x7f5c6403f310 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f5c7801c270 tx=0x7f5c780a6000 comp rx=0 tx=0).stop 2024-09-11T05:22:41.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 10 monclient: shutdown 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8413d640 msgr2=0x7f5c8413ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 --2- 172.21.15.138:0/1485482184 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5c8413d640 0x7f5c8413ca30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00bd90 tx=0x7f5c7c009eb0 comp rx=0 tx=0).stop 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c82d9d640 1 -- 172.21.15.138:0/1485482184 reap_dead start 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c82d9d640 1 -- 172.21.15.138:0/1485482184 reap_dead start 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 1 -- 172.21.15.138:0/1485482184 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(616..616 src has 520..616) ==== 6014+0+0 (unknown 0 0 0) 0x7f5c7c050870 con 0x7f5c8413d640 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 10 client.84494.objecter ms_dispatch 0x7f5c84000ba0 osd_map(616..616 src has 520..616) 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c72ffd640 10 librados: disconnected, discarding osd_map(616..616 src has 520..616) 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 shutdown_connections 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 >> 172.21.15.138:0/1485482184 conn(0x7f5c840ab830 msgr2=0x7f5c8413e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 shutdown_connections 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 -- 172.21.15.138:0/1485482184 wait complete. 2024-09-11T05:22:41.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:22:41.979+0000 7f5c88d0d640 1 librados: shutdown 2024-09-11T05:22:42.000 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:22:42.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:24:12.001 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T05:24:12.002 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T05:24:12.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T05:24:12.006 INFO:tasks.ceph.mon.b:Started 2024-09-11T05:24:12.006 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:24:12.006 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:24:12.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient: get_monmap_and_config 2024-09-11T05:24:12.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient: build_initial_monmap 2024-09-11T05:24:12.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient: monmap: 2024-09-11T05:24:12.136 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:12.142510+0000 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:12.142510+0000 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:12.137 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:12.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 1 Processor -- start 2024-09-11T05:24:12.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 1 -- start start 2024-09-11T05:24:12.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient: init 2024-09-11T05:24:12.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:12.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-11T05:24:12.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99bc06e0e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:12.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99bc09eb30 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:12.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f99bc09ef40 0x7f99bc13e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:12.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient(hunting): picked mon.noname-c con 0x7f99bc09ef40 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:12.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99bc09f540 con 0x7f99bc06e0e0 2024-09-11T05:24:12.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99bc09fb00 con 0x7f99bc09eb30 2024-09-11T05:24:12.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:12.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f99bc06e4c0 con 0x7f99bc09ef40 2024-09-11T05:24:12.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c476e640 10 monclient(hunting): authenticate will time out at 1935.579097s 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c250a640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f99bc06e0e0 0x7f99bc09e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:42088/0 (socket says 172.21.15.138:42088) 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c250a640 1 -- 172.21.15.138:0/2023097757 learned_addr learned my addr 172.21.15.138:0/2023097757 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/2023097757 >> v2:172.21.15.190:3300/0 conn(0x7f99bc09ef40 0x7f99bc13e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 10 monclient(hunting): get_auth_request con 0x7f99bc09ef40 auth_method 0 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:12.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 1 -- 172.21.15.138:0/2023097757 >> v2:172.21.15.190:3300/0 conn(0x7f99bc09ef40 msgr2=0x7f99bc13e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:24:12.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 1 -- 172.21.15.138:0/2023097757 >> v2:172.21.15.190:3300/0 conn(0x7f99bc09ef40 msgr2=0x7f99bc13e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:12.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/2023097757 >> v2:172.21.15.190:3300/0 conn(0x7f99bc09ef40 0x7f99bc13e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:12.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.138+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/2023097757 >> v2:172.21.15.190:3300/0 conn(0x7f99bc09ef40 0x7f99bc13e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:12.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 1 -- 172.21.15.138:0/2023097757 >> v1:172.21.15.138:6789/0 conn(0x7f99bc06e0e0 legacy=0x7f99bc09e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:12.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1d09640 1 -- 172.21.15.138:0/2023097757 >> v1:172.21.15.190:6789/0 conn(0x7f99bc09eb30 legacy=0x7f99bc13a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 <== mon.0 v1:172.21.15.138:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1170270204 0 0) 0x7f99bc09f540 con 0x7f99bc06e0e0 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): _negotiate 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): my global_id is 114192 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 --> v1:172.21.15.138:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99a8003530 con 0x7f99bc06e0e0 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 <== mon.0 v1:172.21.15.138:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3866488551 0 0) 0x7f99a8003530 con 0x7f99bc06e0e0 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 >> v2:172.21.15.190:3300/0 conn(0x7f99bc09ef40 msgr2=0x7f99bc13e060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:12.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 --2- 172.21.15.138:0/2023097757 >> v2:172.21.15.190:3300/0 conn(0x7f99bc09ef40 0x7f99bc13e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 >> v1:172.21.15.190:6789/0 conn(0x7f99bc09eb30 legacy=0x7f99bc13a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:12.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: _finish_hunting 0 2024-09-11T05:24:12.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 monclient: found mon.noname-a-legacy 2024-09-11T05:24:12.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:12.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.138:6789/0 2024-09-11T05:24:12.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 --> v1:172.21.15.138:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99bc0a34c0 con 0x7f99bc06e0e0 2024-09-11T05:24:12.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: _finish_auth 0 2024-09-11T05:24:12.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: _check_auth_tickets 2024-09-11T05:24:12.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:12.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 <== mon.0 v1:172.21.15.138:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2314178977 0 0) 0x7f99b80019b0 con 0x7f99bc06e0e0 2024-09-11T05:24:12.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:12.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T05:24:12.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: dump: 2024-09-11T05:24:12.151 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:12.151 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:12.151 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.151 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.152 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:12.152 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:12.152 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:12.152 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:12.152 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:12.152 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:12.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] but i'm connected to v1:172.21.15.138:6789/0, reconnecting 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 >> v1:172.21.15.138:6789/0 conn(0x7f99bc06e0e0 legacy=0x7f99bc09e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8003b30 0x7f99a8003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): picked mon.c con 0x7f99a8003b30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a8004810 0x7f99a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): picked mon.b con 0x7f99a8004810 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99a8005420 0x7f99a800ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): picked mon.a con 0x7f99a8005420 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc06e4c0 con 0x7f99a8005420 2024-09-11T05:24:12.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f99a800c330 con 0x7f99a8004810 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f99a800c530 con 0x7f99a8003b30 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1508640 10 monclient(hunting): _finish_auth 0 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8003b30 0x7f99a8003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99a8005420 0x7f99a800ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient(hunting): get_auth_request con 0x7f99a8005420 auth_method 0 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:12.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 10 monclient(hunting): get_auth_request con 0x7f99a8003b30 auth_method 0 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1d09640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a8004810 0x7f99a8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:12.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c250a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1d09640 10 monclient(hunting): get_auth_request con 0x7f99a8004810 auth_method 0 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1d09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1d09640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c1d09640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient(hunting): handle_auth_done global_id 114192 payload 293 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 1 -- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8003b30 msgr2=0x7f99a8003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8003b30 0x7f99a8003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 1 -- 172.21.15.138:0/2023097757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a8004810 msgr2=0x7f99a8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:12.156 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99a8004810 0x7f99a8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient: _finish_hunting 0 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 1 monclient: found mon.a 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 1 -- 172.21.15.138:0/2023097757 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a8003530 con 0x7f99a8005420 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient: _finish_auth 0 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 10 monclient: _check_auth_tickets 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c2d0b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c476e640 5 monclient: authenticate success, global_id 114192 2024-09-11T05:24:12.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.142+0000 7f99c476e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c2d0b640 1 -- 172.21.15.138:0/2023097757 reap_dead start 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c2d0b640 1 -- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99a8005420 msgr2=0x7f99a800ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99a8005420 0x7f99a800ba50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009fd0 tx=0x7f99b401ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13db8ef517cc6ba server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4009bd0 con 0x7f99a8005420 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 10 monclient: dump: 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:12.158 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b40a5c30 con 0x7f99a8005420 2024-09-11T05:24:12.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 4 set_mon_vals no callback set 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 1 -- 172.21.15.138:0/2023097757 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b40ad440 con 0x7f99a8005420 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c1508640 10 monclient: dump: 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:12.160 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 10 monclient: shutdown 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 1 -- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99a8005420 msgr2=0x7f99a800ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99a8005420 0x7f99a800ba50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009fd0 tx=0x7f99b401ce20 comp rx=0 tx=0).stop 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 1 -- 172.21.15.138:0/2023097757 shutdown_connections 2024-09-11T05:24:12.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 1 --2- 172.21.15.138:0/2023097757 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99a8005420 0x7f99a800ba50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 1 -- 172.21.15.138:0/2023097757 >> 172.21.15.138:0/2023097757 conn(0x7f99bc063640 msgr2=0x7f99bc063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 1 -- 172.21.15.138:0/2023097757 shutdown_connections 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.146+0000 7f99c476e640 1 -- 172.21.15.138:0/2023097757 wait complete. 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient: build_initial_monmap 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient: monmap: 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:12.162 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:12.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 librados: starting msgr at 2024-09-11T05:24:12.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 librados: starting objecter 2024-09-11T05:24:12.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 Processor -- start 2024-09-11T05:24:12.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 -- start start 2024-09-11T05:24:12.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 librados: setting wanted keys 2024-09-11T05:24:12.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 librados: calling monclient init 2024-09-11T05:24:12.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient: init 2024-09-11T05:24:12.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:12.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:24:12.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8005420 0x7f99bc120cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient(hunting): picked mon.noname-c con 0x7f99a8005420 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99bc121200 0x7f99bc121640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient(hunting): picked mon.noname-a con 0x7f99bc121200 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99bc1cd740 0x7f99bc1cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient(hunting): picked mon.noname-b con 0x7f99bc1cd740 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc0a12b0 con 0x7f99bc121200 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f99bc09f820 con 0x7f99bc1cd740 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f99bc0a1430 con 0x7f99a8005420 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c476e640 10 monclient(hunting): authenticate will time out at 1935.589300s 2024-09-11T05:24:12.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8005420 0x7f99bc120cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8005420 0x7f99bc120cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:60900/0 (socket says 172.21.15.138:60900) 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 -- 172.21.15.138:0/474157495 learned_addr learned my addr 172.21.15.138:0/474157495 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient(hunting): get_auth_request con 0x7f99a8005420 auth_method 0 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c1d09640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99bc121200 0x7f99bc121640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c1d09640 10 monclient(hunting): get_auth_request con 0x7f99bc121200 auth_method 0 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c1d09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c1d09640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c1d09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:12.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c1d09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c1d09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c1d09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99bc1cd740 0x7f99bc1cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient(hunting): handle_auth_done global_id 84503 payload 824 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 -- 172.21.15.138:0/474157495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99bc1cd740 msgr2=0x7f99bc1cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99bc1cd740 0x7f99bc1cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 -- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99bc121200 msgr2=0x7f99bc121640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99bc121200 0x7f99bc121640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient: _finish_hunting 0 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 monclient: found mon.noname-c 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:12.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 -- 172.21.15.138:0/474157495 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99bc1272b0 con 0x7f99a8005420 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c2d0b640 10 monclient: get_auth_request con 0x7f99bc1cd740 auth_method 0 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c2d0b640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99bc1cd740 0x7f99bc1cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 -- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8005420 msgr2=0x7f99bc120cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99c250a640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8005420 0x7f99bc120cc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f99b80056e0 tx=0x7f99b8012630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.150+0000 7f99b2ffd640 10 client.?.objecter ms_handle_connect 0x7f99a8005420 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 1 -- 172.21.15.138:0/474157495 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b801c070 con 0x7f99a8005420 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:12.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:12.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: dump: 2024-09-11T05:24:12.172 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:12.172 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:12.172 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.172 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.172 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: _finish_auth 0 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 1 -- 172.21.15.138:0/474157495 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b8014750 con 0x7f99a8005420 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 5 monclient: authenticate success, global_id 84503 2024-09-11T05:24:12.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 10 monclient: _renew_subs 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 1 -- 172.21.15.138:0/474157495 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8014b00 con 0x7f99a8005420 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99bc127e10 con 0x7f99a8005420 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b3fff640 4 set_mon_vals no callback set 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 monclient: dump: 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:12.174 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 10 client.84503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 10 monclient: _renew_subs 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f99bc09e110 con 0x7f99a8005420 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 1 librados: init done 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:12.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99bc09e4b0 con 0x7f99a8005420 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 1 -- 172.21.15.138:0/474157495 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f99b8014ca0 con 0x7f99a8005420 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 1 --2- 172.21.15.138:0/474157495 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99a003cea0 0x7f99a003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 1 -- 172.21.15.138:0/474157495 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(678..678 src has 520..678) ==== 5983+0+0 (secure 0 0 0) 0x7f99b8052750 con 0x7f99a8005420 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 10 client.84503.objecter ms_dispatch 0x7f99bc000ba0 osd_map(678..678 src has 520..678) 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 3 client.84503.objecter handle_osd_map got epochs [678,678] > 0 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 3 client.84503.objecter handle_osd_map decoding full epoch 678 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99b2ffd640 20 client.84503.objecter dump_active .. 0 homeless 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c1d09640 1 --2- 172.21.15.138:0/474157495 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99a003cea0 0x7f99a003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.154+0000 7f99c1d09640 10 monclient: get_auth_request con 0x7f99a003cea0 auth_method 0 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.158+0000 7f99c1d09640 1 --2- 172.21.15.138:0/474157495 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99a003cea0 0x7f99a003f360 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0a3290 tx=0x7f99ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.158+0000 7f99b2ffd640 10 client.84503.objecter ms_handle_connect 0x7f99a003cea0 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.158+0000 7f99b2ffd640 1 -- 172.21.15.138:0/474157495 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f99b8015d30 con 0x7f99a8005420 2024-09-11T05:24:12.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.158+0000 7f99b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:12.177 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.158+0000 7f99b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:12.392 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.394+0000 7f99c476e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:24:12.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.394+0000 7f99c476e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:12.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.394+0000 7f99c476e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:12.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.394+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f99bc0a2b70 con 0x7f99a8005420 2024-09-11T05:24:12.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.394+0000 7f99b2ffd640 1 -- 172.21.15.138:0/474157495 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f99b801ecd0 con 0x7f99a8005420 2024-09-11T05:24:12.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.394+0000 7f99b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:12.393 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.394+0000 7f99b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:12.393 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.394+0000 7f99c476e640 10 librados: watch_flush enter 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 10 librados: watch_flush exit 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 20 client.84503.objecter shutdown clearing up homeless session... 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 10 client.84503.objecter successfully canceled tick 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99a003cea0 msgr2=0x7f99a003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 --2- 172.21.15.138:0/474157495 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99a003cea0 0x7f99a003f360 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0a3290 tx=0x7f99ac009040 comp rx=0 tx=0).stop 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 10 monclient: shutdown 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:12.397 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8005420 msgr2=0x7f99bc120cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8005420 0x7f99bc120cc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f99b80056e0 tx=0x7f99b8012630 comp rx=0 tx=0).stop 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 shutdown_connections 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 --2- 172.21.15.138:0/474157495 >> v2:172.21.15.190:6800/1562782189 conn(0x7f99a003cea0 0x7f99a003f360 unknown :-1 s=CLOSED pgs=3496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99bc1cd740 0x7f99bc1cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f99bc121200 0x7f99bc121640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 --2- 172.21.15.138:0/474157495 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f99a8005420 0x7f99bc120cc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 >> 172.21.15.138:0/474157495 conn(0x7f99bc063640 msgr2=0x7f99bc134fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 shutdown_connections 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 -- 172.21.15.138:0/474157495 wait complete. 2024-09-11T05:24:12.398 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:12.398+0000 7f99c476e640 1 librados: shutdown 2024-09-11T05:24:15.413 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient: get_monmap_and_config 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient: build_initial_monmap 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient: monmap: 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:15.547150+0000 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:15.547150+0000 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:15.541 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:15.542 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:15.542 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:15.542 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:15.542 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:15.542 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:15.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 1 Processor -- start 2024-09-11T05:24:15.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 1 -- start start 2024-09-11T05:24:15.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient: init 2024-09-11T05:24:15.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:15.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-11T05:24:15.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed6c06e0e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:15.543 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed6c136e30 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fed6c136170 0x7fed6c13ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient(hunting): picked mon.noname-a con 0x7fed6c136170 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed6c0a3270 con 0x7fed6c136e30 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed6c0a0af0 con 0x7fed6c06e0e0 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fed6c06e4c0 con 0x7fed6c136170 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed72450640 10 monclient(hunting): authenticate will time out at 1938.983287s 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fed6c136170 0x7fed6c13ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fed6c136170 0x7fed6c13ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:35650/0 (socket says 172.21.15.138:35650) 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 1 -- 172.21.15.138:0/1220253380 learned_addr learned my addr 172.21.15.138:0/1220253380 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 10 monclient(hunting): get_auth_request con 0x7fed6c136170 auth_method 0 2024-09-11T05:24:15.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:15.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:15.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:15.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:15.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:15.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.542+0000 7fed709ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:15.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed6b7fe640 1 -- 172.21.15.138:0/1220253380 >> v1:172.21.15.138:6789/0 conn(0x7fed6c136e30 legacy=0x7fed6c13b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:15.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed709ed640 10 monclient(hunting): handle_auth_done global_id 114198 payload 274 2024-09-11T05:24:15.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed709ed640 1 -- 172.21.15.138:0/1220253380 >> v1:172.21.15.190:6789/0 conn(0x7fed6c06e0e0 legacy=0x7fed6c136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:15.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed709ed640 1 -- 172.21.15.138:0/1220253380 >> v1:172.21.15.138:6789/0 conn(0x7fed6c136e30 legacy=0x7fed6c13b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:15.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed709ed640 10 monclient: _finish_hunting 0 2024-09-11T05:24:15.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed709ed640 1 monclient: found mon.noname-a 2024-09-11T05:24:15.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed709ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:15.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed709ed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:15.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.546+0000 7fed709ed640 1 -- 172.21.15.138:0/1220253380 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c0a2f90 con 0x7fed6c136170 2024-09-11T05:24:15.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed709ed640 1 -- 172.21.15.138:0/1220253380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 msgr2=0x7fed6c13ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:15.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed709ed640 1 --2- 172.21.15.138:0/1220253380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c13ec00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fed640098f0 tx=0x7fed6401c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554fe2a00a10c05b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:15.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 1 -- 172.21.15.138:0/1220253380 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed640b0070 con 0x7fed6c136170 2024-09-11T05:24:15.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:15.549 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: dump: 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: _finish_auth 0 2024-09-11T05:24:15.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: _check_auth_tickets 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 1 -- 172.21.15.138:0/1220253380 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed640ab600 con 0x7fed6c136170 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 1 -- 172.21.15.138:0/1220253380 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed640a4e60 con 0x7fed6c136170 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed6affd640 10 monclient: dump: 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:15.551 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:15.552 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:15.552 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:15.552 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:15.552 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:15.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 5 monclient: authenticate success, global_id 114198 2024-09-11T05:24:15.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:15.552 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 4 set_mon_vals no callback set 2024-09-11T05:24:15.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 10 monclient: shutdown 2024-09-11T05:24:15.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:15.556 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 1 -- 172.21.15.138:0/1220253380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 msgr2=0x7fed6c13ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 1 --2- 172.21.15.138:0/1220253380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c13ec00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fed640098f0 tx=0x7fed6401c2c0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 1 -- 172.21.15.138:0/1220253380 shutdown_connections 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 1 --2- 172.21.15.138:0/1220253380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c13ec00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 1 -- 172.21.15.138:0/1220253380 >> 172.21.15.138:0/1220253380 conn(0x7fed6c063640 msgr2=0x7fed6c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 1 -- 172.21.15.138:0/1220253380 shutdown_connections 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.550+0000 7fed72450640 1 -- 172.21.15.138:0/1220253380 wait complete. 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient: build_initial_monmap 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient: monmap: 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:15.557 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 librados: starting msgr at 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 librados: starting objecter 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 Processor -- start 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 -- start start 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 librados: setting wanted keys 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 librados: calling monclient init 2024-09-11T05:24:15.558 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient: init 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fed6c06e0e0 0x7fed6c1376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient(hunting): picked mon.noname-c con 0x7fed6c06e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient(hunting): picked mon.noname-a con 0x7fed6c136170 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6c136e30 0x7fed6c1ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient(hunting): picked mon.noname-b con 0x7fed6c136e30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c141cf0 con 0x7fed6c136170 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c141b70 con 0x7fed6c136e30 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:15.559 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fed6c141e70 con 0x7fed6c06e0e0 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed72450640 10 monclient(hunting): authenticate will time out at 1938.993251s 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fed6c06e0e0 0x7fed6c1376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fed6c06e0e0 0x7fed6c1376d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:60910/0 (socket says 172.21.15.138:60910) 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 -- 172.21.15.138:0/1303866675 learned_addr learned my addr 172.21.15.138:0/1303866675 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient(hunting): get_auth_request con 0x7fed6c06e0e0 auth_method 0 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 10 monclient(hunting): get_auth_request con 0x7fed6c136170 auth_method 0 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:15.560 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 1 -- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 msgr2=0x7fed6c137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 1 -- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 msgr2=0x7fed6c137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6b7fe640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c137c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient(hunting): handle_auth_done global_id 84506 payload 824 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed709ed640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6c136e30 0x7fed6c1ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:15.561 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 -- 172.21.15.138:0/1303866675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6c136e30 msgr2=0x7fed6c1ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6c136e30 0x7fed6c1ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 -- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 msgr2=0x7fed6c137c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c137c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 monclient: found mon.noname-c 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.554+0000 7fed6bfff640 1 -- 172.21.15.138:0/1303866675 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c0a6850 con 0x7fed6c06e0e0 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed6bfff640 1 -- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fed6c06e0e0 msgr2=0x7fed6c1376d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed6bfff640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fed6c06e0e0 0x7fed6c1376d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00b410 tx=0x7fed5c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 client.?.objecter ms_handle_connect 0x7fed6c06e0e0 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:15.562 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 1 -- 172.21.15.138:0/1303866675 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c015030 con 0x7fed6c06e0e0 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: dump: 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:15.563 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: _finish_auth 0 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 1 -- 172.21.15.138:0/1303866675 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed5c004020 con 0x7fed6c06e0e0 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 1 -- 172.21.15.138:0/1303866675 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c004410 con 0x7fed6c06e0e0 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 5 monclient: authenticate success, global_id 84506 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 10 monclient: _renew_subs 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed6c09e110 con 0x7fed6c06e0e0 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed6a7fc640 4 set_mon_vals no callback set 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:15.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 monclient: dump: 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:15.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 10 client.84506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 10 monclient: _renew_subs 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed6c1397c0 con 0x7fed6c06e0e0 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 1 librados: init done 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed6c136720 con 0x7fed6c06e0e0 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 1 -- 172.21.15.138:0/1303866675 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fed5c022060 con 0x7fed6c06e0e0 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 1 --2- 172.21.15.138:0/1303866675 >> v2:172.21.15.190:6800/1562782189 conn(0x7fed4c03cef0 0x7fed4c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 1 -- 172.21.15.138:0/1303866675 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(682..682 src has 520..682) ==== 5949+0+0 (secure 0 0 0) 0x7fed5c04f6f0 con 0x7fed6c06e0e0 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 10 client.84506.objecter ms_dispatch 0x7fed6c000ba0 osd_map(682..682 src has 520..682) 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 3 client.84506.objecter handle_osd_map got epochs [682,682] > 0 2024-09-11T05:24:15.566 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 3 client.84506.objecter handle_osd_map decoding full epoch 682 2024-09-11T05:24:15.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed68ff9640 20 client.84506.objecter dump_active .. 0 homeless 2024-09-11T05:24:15.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed6b7fe640 1 --2- 172.21.15.138:0/1303866675 >> v2:172.21.15.190:6800/1562782189 conn(0x7fed4c03cef0 0x7fed4c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:15.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.558+0000 7fed6b7fe640 10 monclient: get_auth_request con 0x7fed4c03cef0 auth_method 0 2024-09-11T05:24:15.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.562+0000 7fed6b7fe640 1 --2- 172.21.15.138:0/1303866675 >> v2:172.21.15.190:6800/1562782189 conn(0x7fed4c03cef0 0x7fed4c03f3b0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7fed6c130b20 tx=0x7fed54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:15.567 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.562+0000 7fed68ff9640 10 client.84506.objecter ms_handle_connect 0x7fed4c03cef0 2024-09-11T05:24:15.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.562+0000 7fed68ff9640 1 -- 172.21.15.138:0/1303866675 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fed5c003dc0 con 0x7fed6c06e0e0 2024-09-11T05:24:15.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.562+0000 7fed68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:15.568 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.562+0000 7fed68ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:15.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.794+0000 7fed72450640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:24:15.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.794+0000 7fed72450640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:15.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.794+0000 7fed72450640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:15.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.794+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fed6c13e020 con 0x7fed6c06e0e0 2024-09-11T05:24:15.795 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed68ff9640 1 -- 172.21.15.138:0/1303866675 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fed5c003980 con 0x7fed6c06e0e0 2024-09-11T05:24:15.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:15.796 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed68ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:15.796 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":95,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:24:15.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 10 librados: watch_flush enter 2024-09-11T05:24:15.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 10 librados: watch_flush exit 2024-09-11T05:24:15.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 20 client.84506.objecter shutdown clearing up homeless session... 2024-09-11T05:24:15.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 10 client.84506.objecter successfully canceled tick 2024-09-11T05:24:15.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 >> v2:172.21.15.190:6800/1562782189 conn(0x7fed4c03cef0 msgr2=0x7fed4c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:15.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 --2- 172.21.15.138:0/1303866675 >> v2:172.21.15.190:6800/1562782189 conn(0x7fed4c03cef0 0x7fed4c03f3b0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7fed6c130b20 tx=0x7fed54009040 comp rx=0 tx=0).stop 2024-09-11T05:24:15.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 10 monclient: shutdown 2024-09-11T05:24:15.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fed6c06e0e0 msgr2=0x7fed6c1376d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fed6c06e0e0 0x7fed6c1376d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00b410 tx=0x7fed5c00b8d0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 shutdown_connections 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 --2- 172.21.15.138:0/1303866675 >> v2:172.21.15.190:6800/1562782189 conn(0x7fed4c03cef0 0x7fed4c03f3b0 unknown :-1 s=CLOSED pgs=3497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed6c136e30 0x7fed6c1ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fed6c136170 0x7fed6c137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 --2- 172.21.15.138:0/1303866675 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fed6c06e0e0 0x7fed6c1376d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 >> 172.21.15.138:0/1303866675 conn(0x7fed6c063640 msgr2=0x7fed6c132d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 shutdown_connections 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 -- 172.21.15.138:0/1303866675 wait complete. 2024-09-11T05:24:15.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:15.798+0000 7fed72450640 1 librados: shutdown 2024-09-11T05:24:18.815 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:24:18.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient: get_monmap_and_config 2024-09-11T05:24:18.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient: build_initial_monmap 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient: monmap: 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:18.951778+0000 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:18.951778+0000 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:18.946 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:18.947 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:18.947 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:18.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 1 Processor -- start 2024-09-11T05:24:18.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 1 -- start start 2024-09-11T05:24:18.949 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient: init 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd3806e0e0 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbd38136e30 0x7fbd3813b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbd38136e30 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3813da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbd38136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd380a3270 con 0x7fbd3806e0e0 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fbd3806e4c0 con 0x7fbd38136e30 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbd380a0af0 con 0x7fbd38136170 2024-09-11T05:24:18.950 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.946+0000 7fbd3e7f8640 10 monclient(hunting): authenticate will time out at 1942.387933s 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbd38136e30 0x7fbd3813b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3813da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fbd38136e30 0x7fbd3813b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49196/0 (socket says 172.21.15.138:49196) 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 1 -- 172.21.15.138:0/662957896 learned_addr learned my addr 172.21.15.138:0/662957896 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 10 monclient(hunting): get_auth_request con 0x7fbd38136e30 auth_method 0 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient(hunting): get_auth_request con 0x7fbd38136170 auth_method 0 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:18.951 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient(hunting): handle_auth_done global_id 84509 payload 274 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 1 -- 172.21.15.138:0/662957896 >> v2:172.21.15.138:3300/0 conn(0x7fbd38136e30 msgr2=0x7fbd3813b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 1 --2- 172.21.15.138:0/662957896 >> v2:172.21.15.138:3300/0 conn(0x7fbd38136e30 0x7fbd3813b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 1 -- 172.21.15.138:0/662957896 >> v1:172.21.15.138:6789/0 conn(0x7fbd3806e0e0 legacy=0x7fbd38136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient: _finish_hunting 0 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 1 monclient: found mon.noname-b 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:24:18.952 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.950+0000 7fbd3cd95640 1 -- 172.21.15.138:0/662957896 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd380a2f90 con 0x7fbd38136170 2024-09-11T05:24:18.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3cd95640 1 -- 172.21.15.138:0/662957896 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 msgr2=0x7fbd3813da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:18.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3cd95640 1 --2- 172.21.15.138:0/662957896 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3813da20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbd300097c0 tx=0x7fbd3001c210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f95a461445f89156 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:18.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 1 -- 172.21.15.138:0/662957896 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd300b0070 con 0x7fbd38136170 2024-09-11T05:24:18.956 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: dump: 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:18.957 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:18.960 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:18.960 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:18.960 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:18.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: _finish_auth 0 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 5 monclient: authenticate success, global_id 84509 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 1 -- 172.21.15.138:0/662957896 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd300ab600 con 0x7fbd38136170 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 1 -- 172.21.15.138:0/662957896 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd300ab9d0 con 0x7fbd38136170 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 4 set_mon_vals no callback set 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:18.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd36ffd640 10 monclient: dump: 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 10 monclient: shutdown 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/662957896 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 msgr2=0x7fbd3813da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:18.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/662957896 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3813da20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbd300097c0 tx=0x7fbd3001c210 comp rx=0 tx=0).stop 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/662957896 shutdown_connections 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/662957896 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3813da20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/662957896 >> v2:172.21.15.138:3300/0 conn(0x7fbd38136e30 0x7fbd3813b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/662957896 >> 172.21.15.138:0/662957896 conn(0x7fbd38063640 msgr2=0x7fbd38063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/662957896 shutdown_connections 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/662957896 wait complete. 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 10 monclient: build_initial_monmap 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 10 monclient: monmap: 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:18.963 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 librados: starting msgr at 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 librados: starting objecter 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 Processor -- start 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 -- start start 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 librados: setting wanted keys 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 librados: calling monclient init 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 10 monclient: init 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:18.964 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd3806e0e0 0x7fbd3812f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbd3806e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.954+0000 7fbd3e7f8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3812ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbd38136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd38136e30 0x7fbd381d0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbd38136e30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fbd38140b10 con 0x7fbd3806e0e0 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbd38140990 con 0x7fbd38136e30 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fbd38140c90 con 0x7fbd38136170 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:18.965 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient(hunting): authenticate will time out at 1942.396056s 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd3806e0e0 0x7fbd3812f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd3806e0e0 0x7fbd3812f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49206/0 (socket says 172.21.15.138:49206) 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 -- 172.21.15.138:0/590002874 learned_addr learned my addr 172.21.15.138:0/590002874 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient(hunting): get_auth_request con 0x7fbd3806e0e0 auth_method 0 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd377fe640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3812ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd377fe640 10 monclient(hunting): get_auth_request con 0x7fbd38136170 auth_method 0 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:18.966 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient(hunting): handle_auth_done global_id 114207 payload 824 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3cd95640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd38136e30 0x7fbd381d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 -- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 msgr2=0x7fbd3812ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3812ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 -- 172.21.15.138:0/590002874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd38136e30 msgr2=0x7fbd381d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd38136e30 0x7fbd381d0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient: _finish_hunting 0 2024-09-11T05:24:18.967 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 monclient: found mon.noname-a 2024-09-11T05:24:18.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:18.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:18.968 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 -- 172.21.15.138:0/590002874 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3813c1c0 con 0x7fbd3806e0e0 2024-09-11T05:24:18.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 -- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd3806e0e0 msgr2=0x7fbd3812f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:18.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd37fff640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd3806e0e0 0x7fbd3812f9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800ba50 tx=0x7fbd2800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 client.?.objecter ms_handle_connect 0x7fbd3806e0e0 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 1 -- 172.21.15.138:0/590002874 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28019070 con 0x7fbd3806e0e0 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: dump: 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:18.971 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: _finish_auth 0 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 1 -- 172.21.15.138:0/590002874 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd2800ce30 con 0x7fbd3806e0e0 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 5 monclient: authenticate success, global_id 114207 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient: _renew_subs 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3813bee0 con 0x7fbd3806e0e0 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:18.972 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 1 -- 172.21.15.138:0/590002874 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28014660 con 0x7fbd3806e0e0 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd34ff9640 10 monclient: dump: 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:18.973 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd367fc640 4 set_mon_vals no callback set 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 client.114207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient: _renew_subs 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd3813b030 con 0x7fbd3806e0e0 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 1 librados: init done 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.958+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd38136720 con 0x7fbd3806e0e0 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd34ff9640 1 -- 172.21.15.138:0/590002874 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fbd280148c0 con 0x7fbd3806e0e0 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd34ff9640 1 --2- 172.21.15.138:0/590002874 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbd1803cf00 0x7fbd1803f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd34ff9640 1 -- 172.21.15.138:0/590002874 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(685..685 src has 520..685) ==== 5949+0+0 (secure 0 0 0) 0x7fbd2804fbe0 con 0x7fbd3806e0e0 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd34ff9640 10 client.114207.objecter ms_dispatch 0x7fbd38000ba0 osd_map(685..685 src has 520..685) 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd34ff9640 3 client.114207.objecter handle_osd_map got epochs [685,685] > 0 2024-09-11T05:24:18.974 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd34ff9640 3 client.114207.objecter handle_osd_map decoding full epoch 685 2024-09-11T05:24:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd34ff9640 20 client.114207.objecter dump_active .. 0 homeless 2024-09-11T05:24:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd377fe640 1 --2- 172.21.15.138:0/590002874 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbd1803cf00 0x7fbd1803f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd377fe640 10 monclient: get_auth_request con 0x7fbd1803cf00 auth_method 0 2024-09-11T05:24:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd34ff9640 10 client.114207.objecter ms_handle_connect 0x7fbd1803cf00 2024-09-11T05:24:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.962+0000 7fbd377fe640 1 --2- 172.21.15.138:0/590002874 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbd1803cf00 0x7fbd1803f3c0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7fbd381d1320 tx=0x7fbd20009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.966+0000 7fbd34ff9640 1 -- 172.21.15.138:0/590002874 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbd28012340 con 0x7fbd3806e0e0 2024-09-11T05:24:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.966+0000 7fbd34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:18.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:18.966+0000 7fbd34ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:19.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.202+0000 7fbd3e7f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:24:19.201 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.202+0000 7fbd3e7f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:19.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.202+0000 7fbd3e7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:19.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.202+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbd3806ccd0 con 0x7fbd3806e0e0 2024-09-11T05:24:19.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.202+0000 7fbd34ff9640 1 -- 172.21.15.138:0/590002874 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fbd2801cdb0 con 0x7fbd3806e0e0 2024-09-11T05:24:19.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.202+0000 7fbd34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:19.202 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.202+0000 7fbd34ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:19.202 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 10 librados: watch_flush enter 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 10 librados: watch_flush exit 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 20 client.114207.objecter shutdown clearing up homeless session... 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 10 client.114207.objecter successfully canceled tick 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbd1803cf00 msgr2=0x7fbd1803f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/590002874 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbd1803cf00 0x7fbd1803f3c0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7fbd381d1320 tx=0x7fbd20009240 comp rx=0 tx=0).stop 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 10 monclient: shutdown 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd3806e0e0 msgr2=0x7fbd3812f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:19.205 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd3806e0e0 0x7fbd3812f9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800ba50 tx=0x7fbd2800bf10 comp rx=0 tx=0).stop 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 shutdown_connections 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/590002874 >> v2:172.21.15.190:6800/1562782189 conn(0x7fbd1803cf00 0x7fbd1803f3c0 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbd38136e30 0x7fbd381d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fbd38136170 0x7fbd3812ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 --2- 172.21.15.138:0/590002874 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fbd3806e0e0 0x7fbd3812f9e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 >> 172.21.15.138:0/590002874 conn(0x7fbd38063640 msgr2=0x7fbd38133d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 shutdown_connections 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 -- 172.21.15.138:0/590002874 wait complete. 2024-09-11T05:24:19.206 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:19.206+0000 7fbd3e7f8640 1 librados: shutdown 2024-09-11T05:24:22.223 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:24:22.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient: get_monmap_and_config 2024-09-11T05:24:22.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient: build_initial_monmap 2024-09-11T05:24:22.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient: monmap: 2024-09-11T05:24:22.352 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:22.358695+0000 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:22.358695+0000 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:22.353 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 1 Processor -- start 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 1 -- start start 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient: init 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2a7806e0e0 0x7f2a78136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a7806e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a78136170 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2a78136c60 0x7f2a780a2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a78136c60 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a780a3270 con 0x7f2a78136170 2024-09-11T05:24:22.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f2a78137070 con 0x7f2a78136c60 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2a7806e4c0 con 0x7f2a7806e0e0 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7f4a5640 10 monclient(hunting): authenticate will time out at 1945.794805s 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2a78136c60 0x7f2a780a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f2a78136c60 0x7f2a780a2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49232/0 (socket says 172.21.15.138:49232) 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 1 -- 172.21.15.138:0/640762172 learned_addr learned my addr 172.21.15.138:0/640762172 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient(hunting): get_auth_request con 0x7f2a78136c60 auth_method 0 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7d241640 1 --2- 172.21.15.138:0/640762172 >> v2:172.21.15.190:3300/0 conn(0x7f2a7806e0e0 0x7f2a78136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7d241640 10 monclient(hunting): get_auth_request con 0x7f2a7806e0e0 auth_method 0 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7d241640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7d241640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7d241640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient(hunting): handle_auth_done global_id 114210 payload 274 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 1 -- 172.21.15.138:0/640762172 >> v2:172.21.15.190:3300/0 conn(0x7f2a7806e0e0 msgr2=0x7f2a78136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 1 --2- 172.21.15.138:0/640762172 >> v2:172.21.15.190:3300/0 conn(0x7f2a7806e0e0 0x7f2a78136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 1 -- 172.21.15.138:0/640762172 >> v1:172.21.15.190:6789/0 conn(0x7f2a78136170 legacy=0x7f2a780a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient: _finish_hunting 0 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 1 monclient: found mon.noname-a 2024-09-11T05:24:22.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.354+0000 7f2a7da42640 1 -- 172.21.15.138:0/640762172 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a780a0af0 con 0x7f2a78136c60 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7da42640 1 -- 172.21.15.138:0/640762172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136c60 msgr2=0x7f2a780a2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7da42640 1 --2- 172.21.15.138:0/640762172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136c60 0x7f2a780a2030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a780647c0 tx=0x7f2a6801c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805fab9c107abe51 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 1 -- 172.21.15.138:0/640762172 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a680b0070 con 0x7f2a78136c60 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: dump: 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.361 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: _finish_auth 0 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: _check_auth_tickets 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 1 -- 172.21.15.138:0/640762172 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a680ab600 con 0x7f2a78136c60 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 1 -- 172.21.15.138:0/640762172 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a680ab9d0 con 0x7f2a78136c60 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 5 monclient: authenticate success, global_id 114210 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 4 set_mon_vals no callback set 2024-09-11T05:24:22.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a77fff640 10 monclient: dump: 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient: shutdown 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:22.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/640762172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136c60 msgr2=0x7f2a780a2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/640762172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136c60 0x7f2a780a2030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2a780647c0 tx=0x7f2a6801c200 comp rx=0 tx=0).stop 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/640762172 shutdown_connections 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/640762172 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136c60 0x7f2a780a2030 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/640762172 >> v2:172.21.15.190:3300/0 conn(0x7f2a7806e0e0 0x7f2a78136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/640762172 >> 172.21.15.138:0/640762172 conn(0x7f2a78063640 msgr2=0x7f2a78063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/640762172 shutdown_connections 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/640762172 wait complete. 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient: build_initial_monmap 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient: monmap: 2024-09-11T05:24:22.364 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 librados: starting msgr at 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 librados: starting objecter 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 Processor -- start 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- start start 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 librados: setting wanted keys 2024-09-11T05:24:22.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 librados: calling monclient init 2024-09-11T05:24:22.366 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient: init 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2a7806e0e0 0x7f2a7813de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a7806e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136170 0x7f2a78138a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a78136170 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a78136c60 0x7f2a78138f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient(hunting): picked mon.noname-b con 0x7f2a78136c60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f2a78140e90 con 0x7f2a78136170 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.370 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2a78140d10 con 0x7f2a78136c60 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f2a78141010 con 0x7f2a7806e0e0 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7f4a5640 10 monclient(hunting): authenticate will time out at 1945.802746s 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136170 0x7f2a78138a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136170 0x7f2a78138a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49238/0 (socket says 172.21.15.138:49238) 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 -- 172.21.15.138:0/1864866376 learned_addr learned my addr 172.21.15.138:0/1864866376 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient(hunting): get_auth_request con 0x7f2a78136170 auth_method 0 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7d241640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2a7806e0e0 0x7f2a7813de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7d241640 10 monclient(hunting): get_auth_request con 0x7f2a7806e0e0 auth_method 0 2024-09-11T05:24:22.371 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7d241640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7d241640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7d241640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7d241640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7d241640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7d241640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7da42640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a78136c60 0x7f2a78138f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7da42640 10 monclient(hunting): get_auth_request con 0x7f2a78136c60 auth_method 0 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7da42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.372 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7da42640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7da42640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient(hunting): handle_auth_done global_id 114213 payload 824 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 -- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2a7806e0e0 msgr2=0x7f2a7813de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2a7806e0e0 0x7f2a7813de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 -- 172.21.15.138:0/1864866376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a78136c60 msgr2=0x7f2a78138f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a78136c60 0x7f2a78138f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient: _finish_hunting 0 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 monclient: found mon.noname-a 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.362+0000 7f2a7ca40640 1 -- 172.21.15.138:0/1864866376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7813a4d0 con 0x7f2a78136170 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7ca40640 1 -- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136170 msgr2=0x7f2a78138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7ca40640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136170 0x7f2a78138a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00c450 tx=0x7f2a6c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.373 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 client.?.objecter ms_handle_connect 0x7f2a78136170 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 1 -- 172.21.15.138:0/1864866376 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c012a40 con 0x7f2a78136170 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: dump: 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.374 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: _finish_auth 0 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 5 monclient: authenticate success, global_id 114213 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 1 -- 172.21.15.138:0/1864866376 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6c017070 con 0x7f2a78136170 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 10 monclient: _renew_subs 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a780a6500 con 0x7f2a78136170 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 1 -- 172.21.15.138:0/1864866376 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c012e40 con 0x7f2a78136170 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 monclient: dump: 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.376 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a777fe640 4 set_mon_vals no callback set 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 10 client.114213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 10 monclient: _renew_subs 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a7813c830 con 0x7f2a78136170 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 1 librados: init done 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 1 -- 172.21.15.138:0/1864866376 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f2a6c01c460 con 0x7f2a78136170 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a78136720 con 0x7f2a78136170 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 1 --2- 172.21.15.138:0/1864866376 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2a6403cea0 0x7f2a6403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 1 -- 172.21.15.138:0/1864866376 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(688..688 src has 520..688) ==== 5962+0+0 (secure 0 0 0) 0x7f2a6c04f9a0 con 0x7f2a78136170 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 10 client.114213.objecter ms_dispatch 0x7f2a78000ba0 osd_map(688..688 src has 520..688) 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 3 client.114213.objecter handle_osd_map got epochs [688,688] > 0 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 3 client.114213.objecter handle_osd_map decoding full epoch 688 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a75ffb640 20 client.114213.objecter dump_active .. 0 homeless 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7d241640 1 --2- 172.21.15.138:0/1864866376 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2a6403cea0 0x7f2a6403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.366+0000 7f2a7d241640 10 monclient: get_auth_request con 0x7f2a6403cea0 auth_method 0 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.370+0000 7f2a7d241640 1 --2- 172.21.15.138:0/1864866376 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2a6403cea0 0x7f2a6403f360 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7f2a700039f0 tx=0x7f2a70009770 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.378 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.370+0000 7f2a75ffb640 10 client.114213.objecter ms_handle_connect 0x7f2a6403cea0 2024-09-11T05:24:22.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.370+0000 7f2a75ffb640 1 -- 172.21.15.138:0/1864866376 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f2a6c01c960 con 0x7f2a78136170 2024-09-11T05:24:22.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.370+0000 7f2a75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:22.379 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.370+0000 7f2a75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:22.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.606+0000 7f2a7f4a5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:24:22.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.606+0000 7f2a7f4a5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:22.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.606+0000 7f2a7f4a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.606+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2a7806ccd0 con 0x7f2a78136170 2024-09-11T05:24:22.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.606+0000 7f2a75ffb640 1 -- 172.21.15.138:0/1864866376 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f2a6c04df00 con 0x7f2a78136170 2024-09-11T05:24:22.606 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.606+0000 7f2a75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:22.607 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.606+0000 7f2a75ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:22.607 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:24:22.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 10 librados: watch_flush enter 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 10 librados: watch_flush exit 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 20 client.114213.objecter shutdown clearing up homeless session... 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 10 client.114213.objecter successfully canceled tick 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2a6403cea0 msgr2=0x7f2a6403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/1864866376 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2a6403cea0 0x7f2a6403f360 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7f2a700039f0 tx=0x7f2a70009770 comp rx=0 tx=0).stop 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 10 monclient: shutdown 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136170 msgr2=0x7f2a78138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136170 0x7f2a78138a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00c450 tx=0x7f2a6c00c910 comp rx=0 tx=0).stop 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 shutdown_connections 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/1864866376 >> v2:172.21.15.190:6800/1562782189 conn(0x7f2a6403cea0 0x7f2a6403f360 unknown :-1 s=CLOSED pgs=3499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2a78136c60 0x7f2a78138f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2a78136170 0x7f2a78138a10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 --2- 172.21.15.138:0/1864866376 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f2a7806e0e0 0x7f2a7813de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 >> 172.21.15.138:0/1864866376 conn(0x7f2a78063640 msgr2=0x7f2a780736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:22.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 shutdown_connections 2024-09-11T05:24:22.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 -- 172.21.15.138:0/1864866376 wait complete. 2024-09-11T05:24:22.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.610+0000 7f2a7f4a5640 1 librados: shutdown 2024-09-11T05:24:22.626 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:24:22.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:24:22.626 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient: get_monmap_and_config 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient: build_initial_monmap 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient: monmap: 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:22.755108+0000 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:22.755108+0000 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:22.749 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:22.750 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:22.750 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.751 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 1 Processor -- start 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 1 -- start start 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient: init 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd0c09da20 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdd0c0a2030 0x7fdd0c13b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient(hunting): picked mon.noname-c con 0x7fdd0c0a2030 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdd0c0a24a0 0x7fdd0c13da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient(hunting): picked mon.noname-a con 0x7fdd0c0a24a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd0c064590 con 0x7fdd0c09da20 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fdd0c09de00 con 0x7fdd0c0a24a0 2024-09-11T05:24:22.752 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdd0c0a2c70 con 0x7fdd0c0a2030 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd147c8640 10 monclient(hunting): authenticate will time out at 1946.191074s 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdd0c0a24a0 0x7fdd0c13da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdd0c0a24a0 0x7fdd0c13da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49242/0 (socket says 172.21.15.138:49242) 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 1 -- 172.21.15.138:0/2266269511 learned_addr learned my addr 172.21.15.138:0/2266269511 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient(hunting): get_auth_request con 0x7fdd0c0a24a0 auth_method 0 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.753 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd11d63640 1 --2- 172.21.15.138:0/2266269511 >> v2:172.21.15.190:3300/0 conn(0x7fdd0c0a2030 0x7fdd0c13b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd11d63640 10 monclient(hunting): get_auth_request con 0x7fdd0c0a2030 auth_method 0 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd11d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd11d63640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd11d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient(hunting): handle_auth_done global_id 114216 payload 274 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 1 -- 172.21.15.138:0/2266269511 >> v2:172.21.15.190:3300/0 conn(0x7fdd0c0a2030 msgr2=0x7fdd0c13b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 1 --2- 172.21.15.138:0/2266269511 >> v2:172.21.15.190:3300/0 conn(0x7fdd0c0a2030 0x7fdd0c13b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 1 -- 172.21.15.138:0/2266269511 >> v1:172.21.15.190:6789/0 conn(0x7fdd0c09da20 legacy=0x7fdd0c0a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient: _finish_hunting 0 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 1 monclient: found mon.noname-a 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.754 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.750+0000 7fdd12d65640 1 -- 172.21.15.138:0/2266269511 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0c0a2f50 con 0x7fdd0c0a24a0 2024-09-11T05:24:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 1 -- 172.21.15.138:0/2266269511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c0a24a0 msgr2=0x7fdd0c13da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.755 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 1 --2- 172.21.15.138:0/2266269511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c0a24a0 0x7fdd0c13da70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c09e050 tx=0x7fdd0401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a00fd6f0614dbd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 1 -- 172.21.15.138:0/2266269511 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040a5750 con 0x7fdd0c0a24a0 2024-09-11T05:24:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:22.756 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: dump: 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: _finish_auth 0 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: _check_auth_tickets 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:22.757 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 1 -- 172.21.15.138:0/2266269511 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd040a58f0 con 0x7fdd0c0a24a0 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 1 -- 172.21.15.138:0/2266269511 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040a5cc0 con 0x7fdd0c0a24a0 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 5 monclient: authenticate success, global_id 114216 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 4 set_mon_vals no callback set 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11562640 10 monclient: dump: 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.758 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient: shutdown 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- 172.21.15.138:0/2266269511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c0a24a0 msgr2=0x7fdd0c13da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 --2- 172.21.15.138:0/2266269511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c0a24a0 0x7fdd0c13da70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c09e050 tx=0x7fdd0401c8b0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- 172.21.15.138:0/2266269511 shutdown_connections 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 --2- 172.21.15.138:0/2266269511 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c0a24a0 0x7fdd0c13da70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 --2- 172.21.15.138:0/2266269511 >> v2:172.21.15.190:3300/0 conn(0x7fdd0c0a2030 0x7fdd0c13b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- 172.21.15.138:0/2266269511 >> 172.21.15.138:0/2266269511 conn(0x7fdd0c063640 msgr2=0x7fdd0c063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- 172.21.15.138:0/2266269511 shutdown_connections 2024-09-11T05:24:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- 172.21.15.138:0/2266269511 wait complete. 2024-09-11T05:24:22.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient: build_initial_monmap 2024-09-11T05:24:22.765 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient: monmap: 2024-09-11T05:24:22.765 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:22.765 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:22.765 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 librados: starting msgr at 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 librados: starting objecter 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 Processor -- start 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- start start 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 librados: setting wanted keys 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 librados: calling monclient init 2024-09-11T05:24:22.766 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient: init 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09da20 0x7fdd0c073d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient(hunting): picked mon.noname-a con 0x7fdd0c09da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd0c0a2030 0x7fdd0c13b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient(hunting): picked mon.noname-b con 0x7fdd0c0a2030 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd0c0a24a0 0x7fdd0c13bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient(hunting): picked mon.noname-c con 0x7fdd0c0a24a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c140b60 con 0x7fdd0c09da20 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c1409e0 con 0x7fdd0c0a2030 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdd0c140ce0 con 0x7fdd0c0a24a0 2024-09-11T05:24:22.767 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd147c8640 10 monclient(hunting): authenticate will time out at 1946.199164s 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09da20 0x7fdd0c073d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09da20 0x7fdd0c073d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49254/0 (socket says 172.21.15.138:49254) 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 -- 172.21.15.138:0/1135427769 learned_addr learned my addr 172.21.15.138:0/1135427769 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient(hunting): get_auth_request con 0x7fdd0c09da20 auth_method 0 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd0c0a24a0 0x7fdd0c13bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 10 monclient(hunting): get_auth_request con 0x7fdd0c0a24a0 auth_method 0 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.768 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11d63640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd0c0a2030 0x7fdd0c13b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11d63640 10 monclient(hunting): get_auth_request con 0x7fdd0c0a2030 auth_method 0 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11d63640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd11d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient(hunting): handle_auth_done global_id 114219 payload 824 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd0c0a24a0 msgr2=0x7fdd0c13bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd0c0a24a0 0x7fdd0c13bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.769 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd0c0a2030 msgr2=0x7fdd0c13b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdd0c0a2030 0x7fdd0c13b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient: _finish_hunting 0 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 monclient: found mon.noname-a 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12564640 1 -- 172.21.15.138:0/1135427769 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0c09e5b0 con 0x7fdd0c09da20 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.758+0000 7fdd12d65640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdd0c0a24a0 0x7fdd0c13bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd12564640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09da20 msgr2=0x7fdd0c073d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd12564640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09da20 0x7fdd0c073d10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800c450 tx=0x7fdd0800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 client.?.objecter ms_handle_connect 0x7fdd0c09da20 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd080119a0 con 0x7fdd0c09da20 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.770 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: dump: 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: _finish_auth 0 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 5 monclient: authenticate success, global_id 114219 2024-09-11T05:24:22.771 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 10 monclient: _renew_subs 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd08017070 con 0x7fdd0c09da20 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0c130180 con 0x7fdd0c09da20 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08011da0 con 0x7fdd0c09da20 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 10 client.114219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd10d61640 4 set_mon_vals no callback set 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 monclient: dump: 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.772 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.773 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.773 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.773 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.773 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 10 monclient: _renew_subs 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0c09e110 con 0x7fdd0c09da20 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 1 librados: init done 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09fe10 0x7fdd0c13c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.774 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c0a1920 con 0x7fdd0c09fe10 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09fe10 0x7fdd0c13c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 10 monclient: get_auth_request con 0x7fdd0c09fe10 auth_method 0 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdd0801c460 con 0x7fdd0c09da20 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 10 monclient(hunting): handle_auth_done global_id 114222 payload 824 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd11d63640 1 -- 172.21.15.138:0/1135427769 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd0c0008e0 con 0x7fdd0c09fe10 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 1 --2- 172.21.15.138:0/1135427769 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcf003ced0 0x7fdcf003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.775 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(689..689 src has 520..689) ==== 5576+0+0 (secure 0 0 0) 0x7fdd0804f810 con 0x7fdd0c09da20 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 10 client.114219.objecter ms_dispatch 0x7fdd0c000ba0 osd_map(689..689 src has 520..689) 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 3 client.114219.objecter handle_osd_map got epochs [689,689] > 0 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 3 client.114219.objecter handle_osd_map decoding full epoch 689 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd02ffd640 20 client.114219.objecter dump_active .. 0 homeless 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.762+0000 7fdd12d65640 1 --2- 172.21.15.138:0/1135427769 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcf003ced0 0x7fdcf003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd12d65640 10 monclient: get_auth_request con 0x7fdcf003ced0 auth_method 0 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 10 client.114219.objecter ms_handle_connect 0x7fdd0c09fe10 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 10 client.114219.objecter resend_mon_ops 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd11d63640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09fe10 msgr2=0x7fdd0c13c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.776 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd11d63640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09fe10 0x7fdd0c13c1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00f4e0 tx=0x7fdcfc00f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc004230 con 0x7fdd0c09fe10 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 10 monclient: dump: 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.777 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdcfc004230 con 0x7fdd0c09fe10 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09fe10 msgr2=0x7fdd0c13c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09fe10 0x7fdd0c13c1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00f4e0 tx=0x7fdcfc00f9a0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09fe10 msgr2=0x7fdd0c13c1a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09fe10 0x7fdd0c13c1a0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00f4e0 tx=0x7fdcfc00f9a0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.778 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd12d65640 1 --2- 172.21.15.138:0/1135427769 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcf003ced0 0x7fdcf003f390 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7fdd040ab000 tx=0x7fdd040a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.779 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.766+0000 7fdd02ffd640 10 client.114219.objecter ms_handle_connect 0x7fdcf003ced0 2024-09-11T05:24:22.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:24:22.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c133df0 0x7fdd0c134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c073590 con 0x7fdd0c133df0 2024-09-11T05:24:22.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c133df0 0x7fdd0c134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 10 monclient: get_auth_request con 0x7fdd0c133df0 auth_method 0 2024-09-11T05:24:22.781 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 10 monclient(hunting): handle_auth_done global_id 114225 payload 824 2024-09-11T05:24:22.782 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 1 -- 172.21.15.138:0/1135427769 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdd0c13d020 con 0x7fdd0c133df0 2024-09-11T05:24:22.783 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c133df0 msgr2=0x7fdd0c134200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 10 client.114219.objecter ms_handle_connect 0x7fdd0c133df0 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 10 client.114219.objecter resend_mon_ops 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd11d63640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c133df0 0x7fdd0c134200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00f4e0 tx=0x7fdcfc0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc003820 con 0x7fdd0c133df0 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 10 monclient: dump: 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.784 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fdcfc003820 con 0x7fdd0c133df0 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c133df0 msgr2=0x7fdd0c134200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c133df0 0x7fdd0c134200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00f4e0 tx=0x7fdcfc0046d0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c133df0 msgr2=0x7fdd0c134200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:22.785 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd02ffd640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c133df0 0x7fdd0c134200 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00f4e0 tx=0x7fdcfc0046d0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 64, 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 134813, 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:24:22.786 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:22.787 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:22.788 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:24:22.789 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:22.790 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:24:22.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:24:22.792 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:22.793 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:24:22.794 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:24:22.795 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:22.796 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:22.797 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:24:22.798 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:22.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 10 librados: watch_flush enter 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 10 librados: watch_flush exit 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 20 client.114219.objecter shutdown clearing up homeless session... 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 10 client.114219.objecter successfully canceled tick 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcf003ced0 msgr2=0x7fdcf003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 --2- 172.21.15.138:0/1135427769 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdcf003ced0 0x7fdcf003f390 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7fdd040ab000 tx=0x7fdd040a7070 comp rx=0 tx=0).stop 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 10 monclient: shutdown 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09da20 msgr2=0x7fdd0c073d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd12d65640 1 -- 172.21.15.138:0/1135427769 reap_dead start 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09da20 0x7fdd0c073d10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800c450 tx=0x7fdd0800c910 comp rx=0 tx=0).stop 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 shutdown_connections 2024-09-11T05:24:22.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 --2- 172.21.15.138:0/1135427769 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdd0c09da20 0x7fdd0c073d10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 >> 172.21.15.138:0/1135427769 conn(0x7fdd0c063640 msgr2=0x7fdd0c133120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 shutdown_connections 2024-09-11T05:24:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 -- 172.21.15.138:0/1135427769 wait complete. 2024-09-11T05:24:22.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.774+0000 7fdd147c8640 1 librados: shutdown 2024-09-11T05:24:22.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:24:22.802 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:24:22.802 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:24:22.914 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.914+0000 7f836d5c7640 10 monclient: get_monmap_and_config 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.914+0000 7f836d5c7640 10 monclient: build_initial_monmap 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.914+0000 7f836d5c7640 10 monclient: monmap: 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:22.921080+0000 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:22.921080+0000 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:22.915 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:22.916 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 1 Processor -- start 2024-09-11T05:24:22.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 1 -- start start 2024-09-11T05:24:22.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient: init 2024-09-11T05:24:22.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:22.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-11T05:24:22.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f83680a48e0 0x7f83680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.918 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient(hunting): picked mon.noname-c con 0x7f83680a48e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f836806fdd0 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f836806ef00 0x7f83680a5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient(hunting): picked mon.noname-b con 0x7f836806ef00 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f836806f450 con 0x7f836806fdd0 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f83680a5740 con 0x7f83680a48e0 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8368070460 con 0x7f836806ef00 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f836d5c7640 10 monclient(hunting): authenticate will time out at 1946.357612s 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f836806ef00 0x7f83680a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83667fc640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f836806fdd0 0x7f8368138a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33284/0 (socket says 172.21.15.138:33284) 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83667fc640 1 -- 172.21.15.138:0/2512025793 learned_addr learned my addr 172.21.15.138:0/2512025793 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:24:22.919 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient(hunting): get_auth_request con 0x7f836806ef00 auth_method 0 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f8366ffd640 1 --2- 172.21.15.138:0/2512025793 >> v2:172.21.15.190:3300/0 conn(0x7f83680a48e0 0x7f83680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f8366ffd640 10 monclient(hunting): get_auth_request con 0x7f83680a48e0 auth_method 0 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f8366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f8366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f8366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient(hunting): handle_auth_done global_id 84521 payload 274 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 1 -- 172.21.15.138:0/2512025793 >> v2:172.21.15.190:3300/0 conn(0x7f83680a48e0 msgr2=0x7f83680a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:22.920 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 1 --2- 172.21.15.138:0/2512025793 >> v2:172.21.15.190:3300/0 conn(0x7f83680a48e0 0x7f83680a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 1 -- 172.21.15.138:0/2512025793 >> v1:172.21.15.138:6789/0 conn(0x7f836806fdd0 legacy=0x7f8368138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:22.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient: _finish_hunting 0 2024-09-11T05:24:22.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 1 monclient: found mon.noname-b 2024-09-11T05:24:22.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:22.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:24:22.921 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.918+0000 7f83677fe640 1 -- 172.21.15.138:0/2512025793 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8368070740 con 0x7f836806ef00 2024-09-11T05:24:22.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f83677fe640 1 -- 172.21.15.138:0/2512025793 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f836806ef00 msgr2=0x7f83680a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f83677fe640 1 --2- 172.21.15.138:0/2512025793 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f836806ef00 0x7f83680a5200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8360009d90 tx=0x7f836001c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11360d5aa2757ef7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 1 -- 172.21.15.138:0/2512025793 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83600b2070 con 0x7f836806ef00 2024-09-11T05:24:22.922 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: dump: 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: _finish_auth 0 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:24:22.923 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 1 -- 172.21.15.138:0/2512025793 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83600a57b0 con 0x7f836806ef00 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 1 -- 172.21.15.138:0/2512025793 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83600a5b80 con 0x7f836806ef00 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 5 monclient: authenticate success, global_id 84521 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 4 set_mon_vals no callback set 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f8365ffb640 10 monclient: dump: 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.924 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 10 monclient: shutdown 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 1 -- 172.21.15.138:0/2512025793 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f836806ef00 msgr2=0x7f83680a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 1 --2- 172.21.15.138:0/2512025793 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f836806ef00 0x7f83680a5200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8360009d90 tx=0x7f836001c830 comp rx=0 tx=0).stop 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 1 -- 172.21.15.138:0/2512025793 shutdown_connections 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 1 --2- 172.21.15.138:0/2512025793 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f836806ef00 0x7f83680a5200 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 1 --2- 172.21.15.138:0/2512025793 >> v2:172.21.15.190:3300/0 conn(0x7f83680a48e0 0x7f83680a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 1 -- 172.21.15.138:0/2512025793 >> 172.21.15.138:0/2512025793 conn(0x7f83680a9590 msgr2=0x7f83680a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 1 -- 172.21.15.138:0/2512025793 shutdown_connections 2024-09-11T05:24:22.925 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.922+0000 7f836d5c7640 1 -- 172.21.15.138:0/2512025793 wait complete. 2024-09-11T05:24:22.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient: build_initial_monmap 2024-09-11T05:24:22.930 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient: monmap: 2024-09-11T05:24:22.930 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:22.930 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:22.930 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:22.930 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:22.930 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 librados: starting msgr at 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 librados: starting objecter 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 Processor -- start 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 -- start start 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 librados: setting wanted keys 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 librados: calling monclient init 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient: init 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:22.931 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f836806ef00 0x7f8368137e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient(hunting): picked mon.noname-a con 0x7f836806ef00 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f836806fdd0 0x7f8368138390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient(hunting): picked mon.noname-b con 0x7f836806fdd0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83680a48e0 0x7f83681d0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient(hunting): picked mon.noname-c con 0x7f83680a48e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f8368143110 con 0x7f836806ef00 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8368142f90 con 0x7f836806fdd0 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f8368143290 con 0x7f83680a48e0 2024-09-11T05:24:22.932 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f836d5c7640 10 monclient(hunting): authenticate will time out at 1946.365550s 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f836806ef00 0x7f8368137e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83680a48e0 0x7f83681d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f836806ef00 0x7f8368137e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49278/0 (socket says 172.21.15.138:49278) 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 1 -- 172.21.15.138:0/2894634043 learned_addr learned my addr 172.21.15.138:0/2894634043 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 10 monclient(hunting): get_auth_request con 0x7f836806ef00 auth_method 0 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient(hunting): get_auth_request con 0x7f83680a48e0 auth_method 0 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.933 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83667fc640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f836806fdd0 0x7f8368138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83667fc640 10 monclient(hunting): get_auth_request con 0x7f836806fdd0 auth_method 0 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83667fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient(hunting): handle_auth_done global_id 84524 payload 824 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f836806fdd0 msgr2=0x7f8368138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f836806fdd0 0x7f8368138390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.934 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f836806ef00 msgr2=0x7f8368137e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f836806ef00 0x7f8368137e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient: _finish_hunting 0 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 1 monclient: found mon.noname-c 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.926+0000 7f83677fe640 1 -- 172.21.15.138:0/2894634043 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836813ea30 con 0x7f83680a48e0 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f83677fe640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83680a48e0 msgr2=0x7f83681d0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f83677fe640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83680a48e0 0x7f83681d0700 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f83600af000 tx=0x7f83600195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 client.?.objecter ms_handle_connect 0x7f83680a48e0 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83600bb070 con 0x7f83680a48e0 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:22.935 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: dump: 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:22.936 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 5 monclient: authenticate success, global_id 84524 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83600ad650 con 0x7f83680a48e0 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83600b2040 con 0x7f83680a48e0 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 10 monclient: _renew_subs 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83680a8850 con 0x7f83680a48e0 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f83657fa640 4 set_mon_vals no callback set 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 10 monclient: dump: 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.937 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 10 client.84524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 10 monclient: _renew_subs 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f836813d6f0 con 0x7f83680a48e0 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 1 librados: init done 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83681d0c40 0x7f83681d1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.938 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f83681d1560 con 0x7f83681d0c40 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83681d0c40 0x7f83681d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 10 monclient: get_auth_request con 0x7f83681d0c40 auth_method 0 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f83677fe640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83680a48e0 0x7f83681d0700 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f83600af000 tx=0x7f83600195d0 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f83600ea000 osd_map(689..689 src has 520..689) 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f83600c0450 con 0x7f83680a48e0 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 10 monclient(hunting): handle_auth_done global_id 84527 payload 824 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f8366ffd640 1 -- 172.21.15.138:0/2894634043 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f836813c6d0 con 0x7f83681d0c40 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f834f7fe640 1 --2- 172.21.15.138:0/2894634043 >> v2:172.21.15.190:6800/1562782189 conn(0x7f835403cf00 0x7f835403f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f83667fc640 1 --2- 172.21.15.138:0/2894634043 >> v2:172.21.15.190:6800/1562782189 conn(0x7f835403cf00 0x7f835403f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.939 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.930+0000 7f83667fc640 10 monclient: get_auth_request con 0x7f835403cf00 auth_method 0 2024-09-11T05:24:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f8366ffd640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83681d0c40 msgr2=0x7f83681d1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 10 client.84524.objecter ms_handle_connect 0x7f83681d0c40 2024-09-11T05:24:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 10 client.84524.objecter resend_mon_ops 2024-09-11T05:24:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f8366ffd640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83681d0c40 0x7f83681d1020 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8358010960 tx=0x7f8358010e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8358018020 con 0x7f83681d0c40 2024-09-11T05:24:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.942 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 10 monclient: dump: 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8358004450 con 0x7f83681d0c40 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:22.943 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83681d0c40 msgr2=0x7f83681d1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83681d0c40 0x7f83681d1020 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8358010960 tx=0x7f8358010e20 comp rx=0 tx=0).stop 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83681d0c40 msgr2=0x7f83681d1020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83681d0c40 0x7f83681d1020 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8358010960 tx=0x7f8358010e20 comp rx=0 tx=0).stop 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f834f7fe640 10 client.84524.objecter ms_handle_connect 0x7f835403cf00 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.934+0000 7f83667fc640 1 --2- 172.21.15.138:0/2894634043 >> v2:172.21.15.190:6800/1562782189 conn(0x7f835403cf00 0x7f835403f3c0 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f835c007fd0 tx=0x7f835c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f836d5c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f836d5c7640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8368071280 0x7f8368071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f836d5c7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f836813d230 con 0x7f8368071280 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8368071280 0x7f8368071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 10 monclient: get_auth_request con 0x7f8368071280 auth_method 0 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:22.944 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 10 monclient(hunting): handle_auth_done global_id 84530 payload 824 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.938+0000 7f8366ffd640 1 -- 172.21.15.138:0/2894634043 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f836806cbf0 con 0x7f8368071280 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f8366ffd640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8368071280 msgr2=0x7f8368071660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f8366ffd640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8368071280 0x7f8368071660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8358010960 tx=0x7f8358004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 client.84524.objecter ms_handle_connect 0x7f8368071280 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 client.84524.objecter resend_mon_ops 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8358018030 con 0x7f8368071280 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 monclient: dump: 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:22.945 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f8358003620 con 0x7f8368071280 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8368071280 msgr2=0x7f8368071660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8368071280 0x7f8368071660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8358010960 tx=0x7f8358004740 comp rx=0 tx=0).stop 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8368071280 msgr2=0x7f8368071660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:22.946 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f8368071280 0x7f8368071660 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8358010960 tx=0x7f8358004740 comp rx=0 tx=0).stop 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 64, 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 506712, 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:24:22.947 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:22.948 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:22.949 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:22.950 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:22.951 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:22.952 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:24:22.953 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.954 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:22.955 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:22.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:24:22.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:22.959 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:22.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:22.959 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:24:22.959 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:22.960 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 10 librados: watch_flush enter 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 10 librados: watch_flush exit 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 20 client.84524.objecter shutdown clearing up homeless session... 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 10 client.84524.objecter successfully canceled tick 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 >> v2:172.21.15.190:6800/1562782189 conn(0x7f835403cf00 msgr2=0x7f835403f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 --2- 172.21.15.138:0/2894634043 >> v2:172.21.15.190:6800/1562782189 conn(0x7f835403cf00 0x7f835403f3c0 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f835c007fd0 tx=0x7f835c009040 comp rx=0 tx=0).stop 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 10 monclient: shutdown 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83680a48e0 msgr2=0x7f83681d0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f83677fe640 1 -- 172.21.15.138:0/2894634043 reap_dead start 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83680a48e0 0x7f83681d0700 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f83600af000 tx=0x7f83600195d0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 1 -- 172.21.15.138:0/2894634043 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(689..689 src has 520..689) ==== 5576+0+0 (unknown 0 0 0) 0x7f83600ea000 con 0x7f83680a48e0 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 client.84524.objecter ms_dispatch 0x7f8368000ba0 osd_map(689..689 src has 520..689) 2024-09-11T05:24:22.961 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f834f7fe640 10 librados: disconnected, discarding osd_map(689..689 src has 520..689) 2024-09-11T05:24:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 shutdown_connections 2024-09-11T05:24:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 --2- 172.21.15.138:0/2894634043 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f83680a48e0 0x7f83681d0700 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 >> 172.21.15.138:0/2894634043 conn(0x7f83680a9590 msgr2=0x7f836809eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 shutdown_connections 2024-09-11T05:24:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 -- 172.21.15.138:0/2894634043 wait complete. 2024-09-11T05:24:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:22.942+0000 7f836d5c7640 1 librados: shutdown 2024-09-11T05:24:22.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:24:22.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:24:22.963 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.098+0000 7fd5fc164640 10 monclient: get_monmap_and_config 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.098+0000 7fd5fc164640 10 monclient: build_initial_monmap 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.098+0000 7fd5fc164640 10 monclient: monmap: 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:23.105340+0000 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:23.105340+0000 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.099 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:23.100 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:23.100 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:23.100 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:23.100 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:23.100 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 1 Processor -- start 2024-09-11T05:24:23.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 1 -- start start 2024-09-11T05:24:23.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient: init 2024-09-11T05:24:23.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:23.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-11T05:24:23.102 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd5f40a5c50 0x7fd5f40a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient(hunting): picked mon.noname-c con 0x7fd5f40a5c50 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd5f40a5460 0x7fd5f40a6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient(hunting): picked mon.noname-a con 0x7fd5f40a5460 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f413ca30 0x7fd5f413ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient(hunting): picked mon.noname-b con 0x7fd5f413ca30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f40a6ab0 con 0x7fd5f40a5460 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f4070930 con 0x7fd5f40a5c50 2024-09-11T05:24:23.103 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f40a6e60 con 0x7fd5f413ca30 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fc164640 10 monclient(hunting): authenticate will time out at 1946.541396s 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd5f40a5460 0x7fd5f40a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f413ca30 0x7fd5f413ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fd5f40a5460 0x7fd5f40a6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49290/0 (socket says 172.21.15.138:49290) 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient(hunting): get_auth_request con 0x7fd5f413ca30 auth_method 0 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 1 -- 172.21.15.138:0/1141085912 learned_addr learned my addr 172.21.15.138:0/1141085912 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 10 monclient(hunting): get_auth_request con 0x7fd5f40a5460 auth_method 0 2024-09-11T05:24:23.104 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient(hunting): handle_auth_done global_id 84533 payload 274 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 1 -- 172.21.15.138:0/1141085912 >> v2:172.21.15.190:3300/0 conn(0x7fd5f40a5c50 msgr2=0x7fd5f40a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/1141085912 >> v2:172.21.15.190:3300/0 conn(0x7fd5f40a5c50 0x7fd5f40a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 1 --2- 172.21.15.138:0/1141085912 >> v2:172.21.15.190:3300/0 conn(0x7fd5f40a5c50 0x7fd5f40a6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 1 -- 172.21.15.138:0/1141085912 >> v2:172.21.15.138:3300/0 conn(0x7fd5f40a5460 msgr2=0x7fd5f40a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:23.105 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 1 --2- 172.21.15.138:0/1141085912 >> v2:172.21.15.138:3300/0 conn(0x7fd5f40a5460 0x7fd5f40a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient: _finish_hunting 0 2024-09-11T05:24:23.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 1 monclient: found mon.noname-b 2024-09-11T05:24:23.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:23.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5fa701640 1 -- 172.21.15.138:0/1141085912 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f40a28f0 con 0x7fd5f413ca30 2024-09-11T05:24:23.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f9f00640 10 monclient: get_auth_request con 0x7fd5f40a5c50 auth_method 0 2024-09-11T05:24:23.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f96ff640 1 --2- 172.21.15.138:0/1141085912 >> v2:172.21.15.138:3300/0 conn(0x7fd5f40a5460 0x7fd5f40a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:24:23.106 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.102+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/1141085912 >> v2:172.21.15.190:3300/0 conn(0x7fd5f40a5c50 0x7fd5f40a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:24:23.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fa701640 1 -- 172.21.15.138:0/1141085912 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f413ca30 msgr2=0x7fd5f413ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:23.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fa701640 1 --2- 172.21.15.138:0/1141085912 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f413ca30 0x7fd5f413ee20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40646a0 tx=0x7fd5ec01c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1492db9919160b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:23.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 1 -- 172.21.15.138:0/1141085912 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0b2070 con 0x7fd5f413ca30 2024-09-11T05:24:23.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.111 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: dump: 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: _finish_auth 0 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: _check_auth_tickets 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:23.112 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 1 -- 172.21.15.138:0/1141085912 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5ec0a5860 con 0x7fd5f413ca30 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 5 monclient: authenticate success, global_id 84533 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 1 -- 172.21.15.138:0/1141085912 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0a5c30 con 0x7fd5f413ca30 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 4 set_mon_vals no callback set 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5f8efe640 10 monclient: dump: 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.113 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 10 monclient: shutdown 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 -- 172.21.15.138:0/1141085912 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f413ca30 msgr2=0x7fd5f413ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 --2- 172.21.15.138:0/1141085912 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f413ca30 0x7fd5f413ee20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40646a0 tx=0x7fd5ec01c7c0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 -- 172.21.15.138:0/1141085912 shutdown_connections 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 --2- 172.21.15.138:0/1141085912 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f413ca30 0x7fd5f413ee20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 --2- 172.21.15.138:0/1141085912 >> v2:172.21.15.138:3300/0 conn(0x7fd5f40a5460 0x7fd5f40a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 --2- 172.21.15.138:0/1141085912 >> v2:172.21.15.190:3300/0 conn(0x7fd5f40a5c50 0x7fd5f40a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 -- 172.21.15.138:0/1141085912 >> 172.21.15.138:0/1141085912 conn(0x7fd5f4063640 msgr2=0x7fd5f4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:23.114 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 -- 172.21.15.138:0/1141085912 shutdown_connections 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 -- 172.21.15.138:0/1141085912 wait complete. 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 10 monclient: build_initial_monmap 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 10 monclient: monmap: 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 librados: starting msgr at 2024-09-11T05:24:23.115 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 librados: starting objecter 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.106+0000 7fd5fc164640 1 Processor -- start 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 -- start start 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 librados: setting wanted keys 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 librados: calling monclient init 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient: init 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd5f40a5460 0x7fd5f41d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient(hunting): picked mon.noname-a con 0x7fd5f40a5460 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f40a5c50 0x7fd5f4139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient(hunting): picked mon.noname-c con 0x7fd5f40a5c50 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f413ca30 0x7fd5f41218a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient(hunting): picked mon.noname-b con 0x7fd5f413ca30 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.116 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f4141d00 con 0x7fd5f40a5460 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f4141b80 con 0x7fd5f413ca30 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f4141e80 con 0x7fd5f40a5c50 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient(hunting): authenticate will time out at 1946.548551s 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd5f40a5460 0x7fd5f41d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f40a5c50 0x7fd5f4139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd5f40a5460 0x7fd5f41d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49294/0 (socket says 172.21.15.138:49294) 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 1 -- 172.21.15.138:0/3983353471 learned_addr learned my addr 172.21.15.138:0/3983353471 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 10 monclient(hunting): get_auth_request con 0x7fd5f40a5460 auth_method 0 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.117 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f40a5c50 0x7fd5f4139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:35610/0 (socket says 172.21.15.138:35610) 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient(hunting): get_auth_request con 0x7fd5f40a5c50 auth_method 0 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient(hunting): handle_auth_done global_id 84536 payload 824 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fa701640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f413ca30 0x7fd5f41218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.118 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f413ca30 msgr2=0x7fd5f41218a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f413ca30 0x7fd5f41218a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd5f40a5460 msgr2=0x7fd5f41d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd5f40a5460 0x7fd5f41d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient: _finish_hunting 0 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 monclient: found mon.noname-c 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 -- 172.21.15.138:0/3983353471 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f40a2440 con 0x7fd5f40a5c50 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fd5f40a5460 0x7fd5f41d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:24:23.119 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f40a5c50 msgr2=0x7fd5f4139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5f96ff640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f40a5c50 0x7fd5f4139a40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40a1e60 tx=0x7fd5e400eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 client.?.objecter ms_handle_connect 0x7fd5f40a5c50 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400cb50 con 0x7fd5f40a5c50 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: dump: 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.120 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: _finish_auth 0 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 5 monclient: authenticate success, global_id 84536 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e400ccf0 con 0x7fd5f40a5c50 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient: _renew_subs 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f413ef20 con 0x7fd5f40a5c50 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4004250 con 0x7fd5f40a5c50 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ea7fc640 10 monclient: dump: 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 client.84536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient: _renew_subs 2024-09-11T05:24:23.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f4123be0 con 0x7fd5f40a5c50 2024-09-11T05:24:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5ebfff640 4 set_mon_vals no callback set 2024-09-11T05:24:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.110+0000 7fd5fc164640 1 librados: init done 2024-09-11T05:24:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5fc164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5fc164640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4123140 0x7fd5f4121de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5fc164640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f40a6030 con 0x7fd5f4123140 2024-09-11T05:24:23.123 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fd5e4023060 con 0x7fd5f40a5c50 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 1 --2- 172.21.15.138:0/3983353471 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd5d803ced0 0x7fd5d803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4123140 0x7fd5f4121de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(690..690 src has 520..690) ==== 5949+0+0 (secure 0 0 0) 0x7fd5e404f980 con 0x7fd5f40a5c50 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 10 client.84536.objecter ms_dispatch 0x7fd5f4000ba0 osd_map(690..690 src has 520..690) 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 3 client.84536.objecter handle_osd_map got epochs [690,690] > 0 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 3 client.84536.objecter handle_osd_map decoding full epoch 690 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 10 monclient: get_auth_request con 0x7fd5f4123140 auth_method 0 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 20 client.84536.objecter dump_active .. 0 homeless 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5fa701640 1 --2- 172.21.15.138:0/3983353471 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd5d803ced0 0x7fd5d803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5fa701640 10 monclient: get_auth_request con 0x7fd5d803ced0 auth_method 0 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_done global_id 124111 payload 824 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 1 -- 172.21.15.138:0/3983353471 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5f40008e0 con 0x7fd5f4123140 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 10 client.84536.objecter ms_handle_connect 0x7fd5f4123140 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4123140 msgr2=0x7fd5f4121de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5ea7fc640 10 client.84536.objecter resend_mon_ops 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.114+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4123140 0x7fd5f4121de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000fc90 tx=0x7fd5f000d590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:23.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 10 client.84536.objecter ms_handle_connect 0x7fd5d803ced0 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5fa701640 1 --2- 172.21.15.138:0/3983353471 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd5d803ced0 0x7fd5d803f390 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4069e60 tx=0x7fd5ec0a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f000fea0 con 0x7fd5f4123140 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 10 monclient: dump: 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.128 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd5f0013850 con 0x7fd5f4123140 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4123140 msgr2=0x7fd5f4121de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4123140 0x7fd5f4121de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000fc90 tx=0x7fd5f000d590 comp rx=0 tx=0).stop 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4123140 msgr2=0x7fd5f4121de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.118+0000 7fd5ea7fc640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4123140 0x7fd5f4121de0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5fc164640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5fc164640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 0x7fd5f4122320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5fc164640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41db360 con 0x7fd5f4139f80 2024-09-11T05:24:23.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 0x7fd5f4122320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5f9f00640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 msgr2=0x7fd5f4122320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T05:24:23.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5f9f00640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 msgr2=0x7fd5f4122320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T05:24:23.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 0x7fd5f4122320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:23.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.122+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 0x7fd5f4122320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:23.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 0x7fd5f4122320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 10 monclient: get_auth_request con 0x7fd5f4139f80 auth_method 0 2024-09-11T05:24:23.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.324 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 10 monclient(hunting): handle_auth_done global_id 124114 payload 824 2024-09-11T05:24:23.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.326+0000 7fd5f9f00640 1 -- 172.21.15.138:0/3983353471 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd5f41db510 con 0x7fd5f4139f80 2024-09-11T05:24:23.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5f9f00640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 msgr2=0x7fd5f4122320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:23.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5f9f00640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 0x7fd5f4122320 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000fc90 tx=0x7fd5f00046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:23.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 10 client.84536.objecter ms_handle_connect 0x7fd5f4139f80 2024-09-11T05:24:23.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 10 client.84536.objecter resend_mon_ops 2024-09-11T05:24:23.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0004460 con 0x7fd5f4139f80 2024-09-11T05:24:23.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:24:23.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 10 monclient: dump: 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.328 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fd5f0004460 con 0x7fd5f4139f80 2024-09-11T05:24:23.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:24:23.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:23.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 msgr2=0x7fd5f4122320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 0x7fd5f4122320 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000fc90 tx=0x7fd5f00046d0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 msgr2=0x7fd5f4122320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:23.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5ea7fc640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd5f4139f80 0x7fd5f4122320 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000fc90 tx=0x7fd5f00046d0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 64, 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 11239, 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:24:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 2, 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:23.331 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:23.332 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:23.333 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:23.334 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:24:23.334 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:23.335 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:23.336 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:23.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:23.338 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:23.339 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:24:23.340 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:24:23.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:23.342 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:24:23.343 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 10 librados: watch_flush enter 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 10 librados: watch_flush exit 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 20 client.84536.objecter shutdown clearing up homeless session... 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 10 client.84536.objecter successfully canceled tick 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd5d803ced0 msgr2=0x7fd5d803f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 --2- 172.21.15.138:0/3983353471 >> v2:172.21.15.190:6800/1562782189 conn(0x7fd5d803ced0 0x7fd5d803f390 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4069e60 tx=0x7fd5ec0a7070 comp rx=0 tx=0).stop 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 10 monclient: shutdown 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f40a5c50 msgr2=0x7fd5f4139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 --2- 172.21.15.138:0/3983353471 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fd5f40a5c50 0x7fd5f4139a40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40a1e60 tx=0x7fd5e400eda0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fa701640 1 -- 172.21.15.138:0/3983353471 reap_dead start 2024-09-11T05:24:23.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fa701640 1 -- 172.21.15.138:0/3983353471 reap_dead start 2024-09-11T05:24:23.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 shutdown_connections 2024-09-11T05:24:23.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 >> 172.21.15.138:0/3983353471 conn(0x7fd5f4063640 msgr2=0x7fd5f413cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:23.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 shutdown_connections 2024-09-11T05:24:23.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 -- 172.21.15.138:0/3983353471 wait complete. 2024-09-11T05:24:23.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.330+0000 7fd5fc164640 1 librados: shutdown 2024-09-11T05:24:23.352 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:24:23.352 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:24:23.352 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:24:23.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.474+0000 7fc5925cd640 10 monclient: get_monmap_and_config 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.474+0000 7fc5925cd640 10 monclient: build_initial_monmap 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.474+0000 7fc5925cd640 10 monclient: monmap: 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:23.481167+0000 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:23.481167+0000 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:23.475 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:23.476 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:23.476 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:23.476 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 1 Processor -- start 2024-09-11T05:24:23.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 1 -- start start 2024-09-11T05:24:23.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient: init 2024-09-11T05:24:23.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:23.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc58c0a3940 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc58c0748b0 0x7fc58c13b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient(hunting): picked mon.noname-c con 0x7fc58c0748b0 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a4430 0x7fc58c13dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient(hunting): picked mon.noname-b con 0x7fc58c0a4430 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc58c09f860 con 0x7fc58c0a3940 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc58c09fe20 con 0x7fc58c0748b0 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc58c09f580 con 0x7fc58c0a4430 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc5925cd640 10 monclient(hunting): authenticate will time out at 1946.917582s 2024-09-11T05:24:23.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc58bfff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fc58c0a3940 0x7fc58c0a3d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33290/0 (socket says 172.21.15.138:33290) 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc58bfff640 1 -- 172.21.15.138:0/3246411048 learned_addr learned my addr 172.21.15.138:0/3246411048 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a4430 0x7fc58c13dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient(hunting): get_auth_request con 0x7fc58c0a4430 auth_method 0 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc58b7fe640 1 --2- 172.21.15.138:0/3246411048 >> v2:172.21.15.190:3300/0 conn(0x7fc58c0748b0 0x7fc58c13b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc58b7fe640 10 monclient(hunting): get_auth_request con 0x7fc58c0748b0 auth_method 0 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc58b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc58b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient(hunting): handle_auth_done global_id 84539 payload 274 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 1 -- 172.21.15.138:0/3246411048 >> v2:172.21.15.190:3300/0 conn(0x7fc58c0748b0 msgr2=0x7fc58c13b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 1 --2- 172.21.15.138:0/3246411048 >> v2:172.21.15.190:3300/0 conn(0x7fc58c0748b0 0x7fc58c13b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 1 -- 172.21.15.138:0/3246411048 >> v1:172.21.15.138:6789/0 conn(0x7fc58c0a3940 legacy=0x7fc58c0a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient: _finish_hunting 0 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 1 monclient: found mon.noname-b 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.478+0000 7fc590b6a640 1 -- 172.21.15.138:0/3246411048 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c09fb40 con 0x7fc58c0a4430 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc590b6a640 1 -- 172.21.15.138:0/3246411048 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a4430 msgr2=0x7fc58c13dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc590b6a640 1 --2- 172.21.15.138:0/3246411048 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a4430 0x7fc58c13dc80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc584009d90 tx=0x7fc58401c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82955b55aceb78b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 1 -- 172.21.15.138:0/3246411048 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5840a6040 con 0x7fc58c0a4430 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: dump: 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.483 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: _finish_auth 0 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: _check_auth_tickets 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 1 -- 172.21.15.138:0/3246411048 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5840af070 con 0x7fc58c0a4430 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 1 -- 172.21.15.138:0/3246411048 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5840aa860 con 0x7fc58c0a4430 2024-09-11T05:24:23.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 5 monclient: authenticate success, global_id 84539 2024-09-11T05:24:23.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:23.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 4 set_mon_vals no callback set 2024-09-11T05:24:23.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:23.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc58affd640 10 monclient: dump: 2024-09-11T05:24:23.485 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.485 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.485 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 10 monclient: shutdown 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 1 -- 172.21.15.138:0/3246411048 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a4430 msgr2=0x7fc58c13dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 1 --2- 172.21.15.138:0/3246411048 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a4430 0x7fc58c13dc80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc584009d90 tx=0x7fc58401c650 comp rx=0 tx=0).stop 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 1 -- 172.21.15.138:0/3246411048 shutdown_connections 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 1 --2- 172.21.15.138:0/3246411048 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a4430 0x7fc58c13dc80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 1 --2- 172.21.15.138:0/3246411048 >> v2:172.21.15.190:3300/0 conn(0x7fc58c0748b0 0x7fc58c13b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 1 -- 172.21.15.138:0/3246411048 >> 172.21.15.138:0/3246411048 conn(0x7fc58c0a8530 msgr2=0x7fc58c0a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 1 -- 172.21.15.138:0/3246411048 shutdown_connections 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.482+0000 7fc5925cd640 1 -- 172.21.15.138:0/3246411048 wait complete. 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: build_initial_monmap 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: monmap: 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:23.487 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:23.488 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 librados: starting msgr at 2024-09-11T05:24:23.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 librados: starting objecter 2024-09-11T05:24:23.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 Processor -- start 2024-09-11T05:24:23.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 -- start start 2024-09-11T05:24:23.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 librados: setting wanted keys 2024-09-11T05:24:23.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 librados: calling monclient init 2024-09-11T05:24:23.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: init 2024-09-11T05:24:23.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:23.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc58c0748b0 0x7fc58c1d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient(hunting): picked mon.noname-b con 0x7fc58c0748b0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a3940 0x7fc58c135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient(hunting): picked mon.noname-c con 0x7fc58c0a3940 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc58c0a4430 0x7fc58c09ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient(hunting): picked mon.noname-a con 0x7fc58c0a4430 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c140d70 con 0x7fc58c0a4430 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c140bf0 con 0x7fc58c0748b0 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fc58c140ef0 con 0x7fc58c0a3940 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:23.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient(hunting): authenticate will time out at 1946.924934s 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a3940 0x7fc58c135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc590b6a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc58c0a4430 0x7fc58c09ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a3940 0x7fc58c135ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:35624/0 (socket says 172.21.15.138:35624) 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 -- 172.21.15.138:0/2037969231 learned_addr learned my addr 172.21.15.138:0/2037969231 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient(hunting): get_auth_request con 0x7fc58c0a3940 auth_method 0 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc590b6a640 10 monclient(hunting): get_auth_request con 0x7fc58c0a4430 auth_method 0 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc590b6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc590b6a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc590b6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc590b6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc590b6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc590b6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58bfff640 1 --2- 172.21.15.138:0/2037969231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc58c0748b0 0x7fc58c1d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58bfff640 10 monclient(hunting): get_auth_request con 0x7fc58c0748b0 auth_method 0 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient(hunting): handle_auth_done global_id 84542 payload 824 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 -- 172.21.15.138:0/2037969231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc58c0748b0 msgr2=0x7fc58c1d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 --2- 172.21.15.138:0/2037969231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc58c0748b0 0x7fc58c1d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 -- 172.21.15.138:0/2037969231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc58c0a4430 msgr2=0x7fc58c09ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 --2- 172.21.15.138:0/2037969231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc58c0a4430 0x7fc58c09ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 monclient: found mon.noname-c 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 -- 172.21.15.138:0/2037969231 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c09f120 con 0x7fc58c0a3940 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 -- 172.21.15.138:0/2037969231 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a3940 msgr2=0x7fc58c135ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58b7fe640 1 --2- 172.21.15.138:0/2037969231 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a3940 0x7fc58c135ac0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d800 tx=0x7fc58000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 client.?.objecter ms_handle_connect 0x7fc58c0a3940 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 1 -- 172.21.15.138:0/2037969231 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580018020 con 0x7fc58c0a3940 2024-09-11T05:24:23.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: dump: 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: _finish_auth 0 2024-09-11T05:24:23.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 1 -- 172.21.15.138:0/2037969231 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5800098b0 con 0x7fc58c0a3940 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 5 monclient: authenticate success, global_id 84542 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: _renew_subs 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58c09e500 con 0x7fc58c0a3940 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 1 -- 172.21.15.138:0/2037969231 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580009c60 con 0x7fc58c0a3940 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc588ff9640 10 monclient: dump: 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:23.498 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 client.84542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: _renew_subs 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58c13cec0 con 0x7fc58c0a3940 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc5925cd640 1 librados: init done 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.486+0000 7fc58a7fc640 4 set_mon_vals no callback set 2024-09-11T05:24:23.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc5925cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc5925cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc5925cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc588ff9640 1 -- 172.21.15.138:0/2037969231 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fc580013050 con 0x7fc58c0a3940 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc58c0a3d20 con 0x7fc58c0a3940 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc588ff9640 1 --2- 172.21.15.138:0/2037969231 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc57803cea0 0x7fc57803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc588ff9640 1 -- 172.21.15.138:0/2037969231 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(690..690 src has 520..690) ==== 5949+0+0 (secure 0 0 0) 0x7fc580050100 con 0x7fc58c0a3940 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc588ff9640 10 client.84542.objecter ms_dispatch 0x7fc58c000ba0 osd_map(690..690 src has 520..690) 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc588ff9640 3 client.84542.objecter handle_osd_map got epochs [690,690] > 0 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc588ff9640 3 client.84542.objecter handle_osd_map decoding full epoch 690 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc588ff9640 20 client.84542.objecter dump_active .. 0 homeless 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc58bfff640 1 --2- 172.21.15.138:0/2037969231 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc57803cea0 0x7fc57803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.490+0000 7fc58bfff640 10 monclient: get_auth_request con 0x7fc57803cea0 auth_method 0 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.494+0000 7fc58bfff640 1 --2- 172.21.15.138:0/2037969231 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc57803cea0 0x7fc57803f360 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7fc57c007fd0 tx=0x7fc57c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:23.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.494+0000 7fc588ff9640 10 client.84542.objecter ms_handle_connect 0x7fc57803cea0 2024-09-11T05:24:23.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.494+0000 7fc588ff9640 1 -- 172.21.15.138:0/2037969231 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fc580022020 con 0x7fc58c0a3940 2024-09-11T05:24:23.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.494+0000 7fc588ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:23.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.494+0000 7fc588ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:23.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.730+0000 7fc5925cd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:24:23.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.730+0000 7fc5925cd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:24:23.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.730+0000 7fc5925cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:23.728 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.730+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc58c13c2c0 con 0x7fc58c0a3940 2024-09-11T05:24:23.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fc588ff9640 1 -- 172.21.15.138:0/2037969231 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc580009e00 con 0x7fc58c0a3940 2024-09-11T05:24:23.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fc588ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:24:23.731 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fc588ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:23.732 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.732 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=11,kv=10,logm=55} crc {auth=2492096145,config=692619927,health=294149932,kv=310889196,logm=663115474}) 2024-09-11T05:24:23.732 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=11,kv=10,logm=56} crc {auth=1991212125,config=692619927,health=889897983,kv=310889196,logm=2745610381}) 2024-09-11T05:24:23.732 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.732 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=11,kv=10,logm=55} crc {auth=2492096145,config=692619927,health=294149932,kv=310889196,logm=663115474}) 2024-09-11T05:24:23.732 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=11,kv=9,logm=57} crc {auth=1515653888,config=692619927,health=736623974,kv=1804961813,logm=3473673679}) 2024-09-11T05:24:23.734 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.735 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2368820640}) 2024-09-11T05:24:23.735 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=532923602}) 2024-09-11T05:24:23.735 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.735 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2368820640}) 2024-09-11T05:24:23.735 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.734+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479725705}) 2024-09-11T05:24:23.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 10 librados: watch_flush enter 2024-09-11T05:24:23.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 10 librados: watch_flush exit 2024-09-11T05:24:23.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 20 client.84542.objecter shutdown clearing up homeless session... 2024-09-11T05:24:23.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 10 client.84542.objecter successfully canceled tick 2024-09-11T05:24:23.736 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc57803cea0 msgr2=0x7fc57803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 --2- 172.21.15.138:0/2037969231 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc57803cea0 0x7fc57803f360 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7fc57c007fd0 tx=0x7fc57c009040 comp rx=0 tx=0).stop 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 10 monclient: shutdown 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a3940 msgr2=0x7fc58c135ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 --2- 172.21.15.138:0/2037969231 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a3940 0x7fc58c135ac0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d800 tx=0x7fc58000dcc0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 shutdown_connections 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 --2- 172.21.15.138:0/2037969231 >> v2:172.21.15.190:6800/1562782189 conn(0x7fc57803cea0 0x7fc57803f360 unknown :-1 s=CLOSED pgs=3503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 --2- 172.21.15.138:0/2037969231 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fc58c0a4430 0x7fc58c09ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 --2- 172.21.15.138:0/2037969231 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fc58c0a3940 0x7fc58c135ac0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 --2- 172.21.15.138:0/2037969231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc58c0748b0 0x7fc58c1d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 >> 172.21.15.138:0/2037969231 conn(0x7fc58c0a8530 msgr2=0x7fc58c0a5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 shutdown_connections 2024-09-11T05:24:23.737 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 -- 172.21.15.138:0/2037969231 wait complete. 2024-09-11T05:24:23.738 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fc5925cd640 1 librados: shutdown 2024-09-11T05:24:23.738 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.738 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181412537}) 2024-09-11T05:24:23.738 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2567149743}) 2024-09-11T05:24:23.738 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.739 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181412537}) 2024-09-11T05:24:23.739 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742296009}) 2024-09-11T05:24:23.739 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.740 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3524363938}) 2024-09-11T05:24:23.740 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=240687755}) 2024-09-11T05:24:23.740 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.740 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.738+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3524363938}) 2024-09-11T05:24:23.740 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1075397354}) 2024-09-11T05:24:23.741 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.741 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572767240}) 2024-09-11T05:24:23.741 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1279383274}) 2024-09-11T05:24:23.741 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.741 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572767240}) 2024-09-11T05:24:23.741 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1851913443}) 2024-09-11T05:24:23.743 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.743 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865773569}) 2024-09-11T05:24:23.743 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1737809499}) 2024-09-11T05:24:23.743 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.743 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865773569}) 2024-09-11T05:24:23.743 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.742+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783919387}) 2024-09-11T05:24:23.744 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.744 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199887581}) 2024-09-11T05:24:23.745 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=121495562}) 2024-09-11T05:24:23.745 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.745 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199887581}) 2024-09-11T05:24:23.745 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4075908361}) 2024-09-11T05:24:23.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254286442}) 2024-09-11T05:24:23.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538653131}) 2024-09-11T05:24:23.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.747 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254286442}) 2024-09-11T05:24:23.747 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=97254773}) 2024-09-11T05:24:23.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=31698000}) 2024-09-11T05:24:23.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627502040}) 2024-09-11T05:24:23.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.749 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=31698000}) 2024-09-11T05:24:23.749 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954903872}) 2024-09-11T05:24:23.750 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.750 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408726171}) 2024-09-11T05:24:23.750 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2947117434}) 2024-09-11T05:24:23.750 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.750 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408726171}) 2024-09-11T05:24:23.750 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018263190}) 2024-09-11T05:24:23.752 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.752 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501464422}) 2024-09-11T05:24:23.752 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058103961}) 2024-09-11T05:24:23.752 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.752 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501464422}) 2024-09-11T05:24:23.752 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1952702829}) 2024-09-11T05:24:23.753 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:24:23.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574793518}) 2024-09-11T05:24:23.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2967919635}) 2024-09-11T05:24:23.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574793518}) 2024-09-11T05:24:23.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3276608174}) 2024-09-11T05:24:23.755 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.755 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362212853}) 2024-09-11T05:24:23.755 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3365864154}) 2024-09-11T05:24:23.755 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.755 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362212853}) 2024-09-11T05:24:23.755 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3025770815}) 2024-09-11T05:24:23.757 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.757 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427846016}) 2024-09-11T05:24:23.757 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197851719}) 2024-09-11T05:24:23.757 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.757 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427846016}) 2024-09-11T05:24:23.758 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1998756185}) 2024-09-11T05:24:23.759 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.759 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462275677}) 2024-09-11T05:24:23.759 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3738941753}) 2024-09-11T05:24:23.759 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.759 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462275677}) 2024-09-11T05:24:23.759 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820659163}) 2024-09-11T05:24:23.760 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.760 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519170109}) 2024-09-11T05:24:23.761 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4120271695}) 2024-09-11T05:24:23.761 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.761 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519170109}) 2024-09-11T05:24:23.761 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=727878262}) 2024-09-11T05:24:23.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499083944}) 2024-09-11T05:24:23.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2723118780}) 2024-09-11T05:24:23.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499083944}) 2024-09-11T05:24:23.763 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1326658619}) 2024-09-11T05:24:23.764 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.764 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2921494810}) 2024-09-11T05:24:23.764 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1379148640}) 2024-09-11T05:24:23.764 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.764 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2921494810}) 2024-09-11T05:24:23.764 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342168798}) 2024-09-11T05:24:23.766 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731141940}) 2024-09-11T05:24:23.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2105194080}) 2024-09-11T05:24:23.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731141940}) 2024-09-11T05:24:23.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667136620}) 2024-09-11T05:24:23.768 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.768 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=601766620}) 2024-09-11T05:24:23.768 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=114539678}) 2024-09-11T05:24:23.768 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.768 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=601766620}) 2024-09-11T05:24:23.768 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3983607139}) 2024-09-11T05:24:23.770 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.770 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594366244}) 2024-09-11T05:24:23.770 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2041403999}) 2024-09-11T05:24:23.770 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.770 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594366244}) 2024-09-11T05:24:23.770 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.770+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3541502564}) 2024-09-11T05:24:23.771 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.772 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312367307}) 2024-09-11T05:24:23.772 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4032176472}) 2024-09-11T05:24:23.772 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.772 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312367307}) 2024-09-11T05:24:23.772 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230927969}) 2024-09-11T05:24:23.773 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.773 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1306547014}) 2024-09-11T05:24:23.773 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=143262516}) 2024-09-11T05:24:23.773 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.773 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1306547014}) 2024-09-11T05:24:23.774 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3361838699}) 2024-09-11T05:24:23.775 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.775 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=768920648}) 2024-09-11T05:24:23.775 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=516767406}) 2024-09-11T05:24:23.775 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.774+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.775 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=768920648}) 2024-09-11T05:24:23.775 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152753451}) 2024-09-11T05:24:23.777 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.777 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685677898}) 2024-09-11T05:24:23.777 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=366658985}) 2024-09-11T05:24:23.777 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.777 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685677898}) 2024-09-11T05:24:23.777 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771574563}) 2024-09-11T05:24:23.778 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.779 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847283692}) 2024-09-11T05:24:23.779 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607562568}) 2024-09-11T05:24:23.779 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.779 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847283692}) 2024-09-11T05:24:23.779 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.778+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324155936}) 2024-09-11T05:24:23.780 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.780 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415984478}) 2024-09-11T05:24:23.780 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3605282443}) 2024-09-11T05:24:23.780 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.781 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415984478}) 2024-09-11T05:24:23.781 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923749660}) 2024-09-11T05:24:23.782 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.782 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115551407}) 2024-09-11T05:24:23.782 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423221173}) 2024-09-11T05:24:23.782 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.782 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115551407}) 2024-09-11T05:24:23.782 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.782+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=583830495}) 2024-09-11T05:24:23.784 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.784 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2999158554}) 2024-09-11T05:24:23.784 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4031129457}) 2024-09-11T05:24:23.784 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.784 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2999158554}) 2024-09-11T05:24:23.784 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2894204713}) 2024-09-11T05:24:23.785 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.785 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364744725}) 2024-09-11T05:24:23.785 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3416261851}) 2024-09-11T05:24:23.785 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.785 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364744725}) 2024-09-11T05:24:23.786 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.786+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2313901321}) 2024-09-11T05:24:23.787 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.787 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10344042}) 2024-09-11T05:24:23.787 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487690528}) 2024-09-11T05:24:23.787 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.787 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10344042}) 2024-09-11T05:24:23.787 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2958265395}) 2024-09-11T05:24:23.788 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.789 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1756419993}) 2024-09-11T05:24:23.789 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3231864458}) 2024-09-11T05:24:23.789 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.789 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1756419993}) 2024-09-11T05:24:23.789 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2970379527}) 2024-09-11T05:24:23.791 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.791 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400189536}) 2024-09-11T05:24:23.791 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2998138524}) 2024-09-11T05:24:23.791 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.791 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400189536}) 2024-09-11T05:24:23.791 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.790+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=274923588}) 2024-09-11T05:24:23.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509531329}) 2024-09-11T05:24:23.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1209624354}) 2024-09-11T05:24:23.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509531329}) 2024-09-11T05:24:23.792 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3039121530}) 2024-09-11T05:24:23.793 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.793 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389719298}) 2024-09-11T05:24:23.794 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274906051}) 2024-09-11T05:24:23.794 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.794 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389719298}) 2024-09-11T05:24:23.794 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.794+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1783332268}) 2024-09-11T05:24:23.795 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.795 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437458436}) 2024-09-11T05:24:23.795 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684820532}) 2024-09-11T05:24:23.795 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.795 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437458436}) 2024-09-11T05:24:23.795 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2061023546}) 2024-09-11T05:24:23.797 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.797 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837596091}) 2024-09-11T05:24:23.797 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745541387}) 2024-09-11T05:24:23.797 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.797 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837596091}) 2024-09-11T05:24:23.797 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3416854968}) 2024-09-11T05:24:23.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965932252}) 2024-09-11T05:24:23.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=635175049}) 2024-09-11T05:24:23.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965932252}) 2024-09-11T05:24:23.799 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.798+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629253039}) 2024-09-11T05:24:23.800 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.800 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553221278}) 2024-09-11T05:24:23.800 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552788676}) 2024-09-11T05:24:23.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553221278}) 2024-09-11T05:24:23.801 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553781262}) 2024-09-11T05:24:23.802 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.802 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998527571}) 2024-09-11T05:24:23.802 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3544292313}) 2024-09-11T05:24:23.802 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.802 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998527571}) 2024-09-11T05:24:23.802 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.802+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1778698507}) 2024-09-11T05:24:23.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6065087}) 2024-09-11T05:24:23.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4050813622}) 2024-09-11T05:24:23.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6065087}) 2024-09-11T05:24:23.804 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351706604}) 2024-09-11T05:24:23.806 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.806 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052857344}) 2024-09-11T05:24:23.806 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4161794448}) 2024-09-11T05:24:23.806 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.806 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052857344}) 2024-09-11T05:24:23.806 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.806+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=113909987}) 2024-09-11T05:24:23.808 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.808 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870013494}) 2024-09-11T05:24:23.808 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1496319287}) 2024-09-11T05:24:23.808 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.808 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870013494}) 2024-09-11T05:24:23.808 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1908646288}) 2024-09-11T05:24:23.809 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.809 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212692383}) 2024-09-11T05:24:23.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=615534971}) 2024-09-11T05:24:23.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212692383}) 2024-09-11T05:24:23.810 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.810+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399801851}) 2024-09-11T05:24:23.811 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.811 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552510962}) 2024-09-11T05:24:23.811 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832813499}) 2024-09-11T05:24:23.811 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.811 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552510962}) 2024-09-11T05:24:23.812 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=217308603}) 2024-09-11T05:24:23.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900938942}) 2024-09-11T05:24:23.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2721598162}) 2024-09-11T05:24:23.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900938942}) 2024-09-11T05:24:23.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=956018049}) 2024-09-11T05:24:23.815 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.815 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374918665}) 2024-09-11T05:24:23.815 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151236095}) 2024-09-11T05:24:23.815 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.814+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.815 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374918665}) 2024-09-11T05:24:23.815 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284771502}) 2024-09-11T05:24:23.816 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.817 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882822688}) 2024-09-11T05:24:23.817 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3049660028}) 2024-09-11T05:24:23.817 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.817 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882822688}) 2024-09-11T05:24:23.817 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2988781598}) 2024-09-11T05:24:23.818 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.818 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343472018}) 2024-09-11T05:24:23.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2632412080}) 2024-09-11T05:24:23.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343472018}) 2024-09-11T05:24:23.819 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.818+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2937372811}) 2024-09-11T05:24:23.820 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.820 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=173383527}) 2024-09-11T05:24:23.820 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299798390}) 2024-09-11T05:24:23.820 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.820 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=173383527}) 2024-09-11T05:24:23.821 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=148329688}) 2024-09-11T05:24:23.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233123319}) 2024-09-11T05:24:23.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646734095}) 2024-09-11T05:24:23.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233123319}) 2024-09-11T05:24:23.822 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.822+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=348205229}) 2024-09-11T05:24:23.824 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.826+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.824 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.826+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150281661}) 2024-09-11T05:24:23.824 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.826+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556029932}) 2024-09-11T05:24:23.824 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.826+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:23.824 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.826+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150281661}) 2024-09-11T05:24:23.824 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:23.826+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329093113}) 2024-09-11T05:24:24.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.022+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.022+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990264987}) 2024-09-11T05:24:24.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.022+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813339624}) 2024-09-11T05:24:24.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.022+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.022+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990264987}) 2024-09-11T05:24:24.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.022+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232324925}) 2024-09-11T05:24:24.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.034+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.034+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430503414}) 2024-09-11T05:24:24.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.034+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2531475647}) 2024-09-11T05:24:24.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.034+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.034+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430503414}) 2024-09-11T05:24:24.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.034+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2129240841}) 2024-09-11T05:24:24.098 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.098+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.098 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.098+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3166710007}) 2024-09-11T05:24:24.098 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.098+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=706199909}) 2024-09-11T05:24:24.098 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.098+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.098 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.098+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3166710007}) 2024-09-11T05:24:24.098 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.098+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3980816067}) 2024-09-11T05:24:24.102 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.102+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.102 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.102+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50644125}) 2024-09-11T05:24:24.102 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.102+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2163064694}) 2024-09-11T05:24:24.102 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.102+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.102 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.102+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50644125}) 2024-09-11T05:24:24.103 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.102+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2853295301}) 2024-09-11T05:24:24.695 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.698+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.696 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.698+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802496995}) 2024-09-11T05:24:24.696 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.698+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2421948649}) 2024-09-11T05:24:24.696 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.698+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.696 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.698+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802496995}) 2024-09-11T05:24:24.696 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.698+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27554824}) 2024-09-11T05:24:24.719 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.722+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.719 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.722+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276342064}) 2024-09-11T05:24:24.719 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.722+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=277307894}) 2024-09-11T05:24:24.719 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.722+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.719 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.722+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276342064}) 2024-09-11T05:24:24.719 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.722+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=314516205}) 2024-09-11T05:24:24.744 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.744 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113679692}) 2024-09-11T05:24:24.744 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505506200}) 2024-09-11T05:24:24.744 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.744 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113679692}) 2024-09-11T05:24:24.744 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890319609}) 2024-09-11T05:24:24.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268818459}) 2024-09-11T05:24:24.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=492925016}) 2024-09-11T05:24:24.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268818459}) 2024-09-11T05:24:24.746 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.746+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1215597106}) 2024-09-11T05:24:24.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666472088}) 2024-09-11T05:24:24.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=828572657}) 2024-09-11T05:24:24.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666472088}) 2024-09-11T05:24:24.748 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4228107858}) 2024-09-11T05:24:24.749 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.749 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250433018}) 2024-09-11T05:24:24.749 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417848010}) 2024-09-11T05:24:24.749 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.749 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250433018}) 2024-09-11T05:24:24.749 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.750+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3149461054}) 2024-09-11T05:24:24.751 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.751 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3779184230}) 2024-09-11T05:24:24.751 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480708812}) 2024-09-11T05:24:24.751 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.751 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3779184230}) 2024-09-11T05:24:24.752 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139408025}) 2024-09-11T05:24:24.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=553937033,mdsmap=1688852554}) 2024-09-11T05:24:24.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3209145160,mdsmap=1688852554}) 2024-09-11T05:24:24.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=553937033,mdsmap=1688852554}) 2024-09-11T05:24:24.754 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.754+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=704206214,mdsmap=1754867906,mgr=4007307362}) 2024-09-11T05:24:24.754 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:24:24.755 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:24:24.758 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.758 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=37,osdmap=19} crc {mdsmap=3140514009,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2659901248,mon_config_key=866245988,monmap=967756510,nvmeofgw=3140514009,osd_metadata=98998854,osd_pg_creating=3649411642,osd_snap=3522664407,osdmap=1434498311}) 2024-09-11T05:24:24.758 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=18} crc {mdsmap=3140514009,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2659901248,mon_config_key=438555161,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3649411642,osd_snap=2764142748,osdmap=2040015173}) 2024-09-11T05:24:24.758 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.758 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=37,osdmap=19} crc {mdsmap=3140514009,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2659901248,mon_config_key=866245988,monmap=967756510,nvmeofgw=3140514009,osd_metadata=98998854,osd_pg_creating=3649411642,osd_snap=3522664407,osdmap=1434498311}) 2024-09-11T05:24:24.758 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.758+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=18} crc {mdsmap=3140514009,mgr=3390704058,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2659901248,mon_config_key=866245988,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=3649411642,osd_snap=3614761055,osdmap=1154774662}) 2024-09-11T05:24:24.761 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.761 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3170437648}) 2024-09-11T05:24:24.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3351771498}) 2024-09-11T05:24:24.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3170437648}) 2024-09-11T05:24:24.762 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.762+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2218149953}) 2024-09-11T05:24:24.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1707889576}) 2024-09-11T05:24:24.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3411438618}) 2024-09-11T05:24:24.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1707889576}) 2024-09-11T05:24:24.767 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.766+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1859888480}) 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient: get_monmap_and_config 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient: build_initial_monmap 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient: monmap: 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:24.887260+0000 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:24.887260+0000 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:24.881 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:24.882 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:24.882 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:24.882 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:24.882 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:24.882 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:24.882 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 1 Processor -- start 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 1 -- start start 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient: init 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f90001355e0 0x7f90001359c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient(hunting): picked mon.noname-c con 0x7f90001355e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9000134dd0 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:24.884 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9000138a60 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9000074990 con 0x7f9000138a60 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90000a2740 con 0x7f9000134dd0 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f90000a2d00 con 0x7f90001355e0 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f9006300640 10 monclient(hunting): authenticate will time out at 1948.323340s 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f900489d640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f9000138a60 0x7f90001406f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33292/0 (socket says 172.21.15.138:33292) 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f900489d640 1 -- 172.21.15.138:0/2456015639 learned_addr learned my addr 172.21.15.138:0/2456015639 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f8ffffff640 1 --2- 172.21.15.138:0/2456015639 >> v2:172.21.15.190:3300/0 conn(0x7f90001355e0 0x7f90001359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f8ffffff640 10 monclient(hunting): get_auth_request con 0x7f90001355e0 auth_method 0 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f8ffffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f8ffffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:24.885 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f8ffffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f8ffffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f8ffffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.882+0000 7f8ffffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.886+0000 7f8ffffff640 10 monclient(hunting): handle_auth_done global_id 124123 payload 274 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.886+0000 7f8ffffff640 1 -- 172.21.15.138:0/2456015639 >> v1:172.21.15.190:6789/0 conn(0x7f9000134dd0 legacy=0x7f9000138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.886+0000 7f8ffffff640 1 -- 172.21.15.138:0/2456015639 >> v1:172.21.15.138:6789/0 conn(0x7f9000138a60 legacy=0x7f90001406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.886+0000 7f8ffffff640 10 monclient: _finish_hunting 0 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.886+0000 7f8ffffff640 1 monclient: found mon.noname-c 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.886+0000 7f8ffffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.886+0000 7f8ffffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:24:24.886 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.886+0000 7f8ffffff640 1 -- 172.21.15.138:0/2456015639 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90000a2a20 con 0x7f90001355e0 2024-09-11T05:24:24.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffffff640 1 -- 172.21.15.138:0/2456015639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90001355e0 msgr2=0x7f90001359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:24.887 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffffff640 1 --2- 172.21.15.138:0/2456015639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90001355e0 0x7f90001359c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00097c0 tx=0x7f8ff001c180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d720d0a47a43ea35 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:24.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 1 -- 172.21.15.138:0/2456015639 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00b0070 con 0x7f90001355e0 2024-09-11T05:24:24.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:24.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:24.888 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: dump: 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: _finish_auth 0 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: _check_auth_tickets 2024-09-11T05:24:24.889 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 1 -- 172.21.15.138:0/2456015639 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ff00ab600 con 0x7f90001355e0 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 1 -- 172.21.15.138:0/2456015639 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00ab9d0 con 0x7f90001355e0 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f8ffeffd640 10 monclient: dump: 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:24.890 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:24.891 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:24.891 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:24.891 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:24.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 5 monclient: authenticate success, global_id 124123 2024-09-11T05:24:24.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:24.891 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 4 set_mon_vals no callback set 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 10 monclient: shutdown 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 -- 172.21.15.138:0/2456015639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90001355e0 msgr2=0x7f90001359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 --2- 172.21.15.138:0/2456015639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90001355e0 0x7f90001359c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00097c0 tx=0x7f8ff001c180 comp rx=0 tx=0).stop 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 -- 172.21.15.138:0/2456015639 shutdown_connections 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 --2- 172.21.15.138:0/2456015639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90001355e0 0x7f90001359c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 -- 172.21.15.138:0/2456015639 >> 172.21.15.138:0/2456015639 conn(0x7f9000063640 msgr2=0x7f9000063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 -- 172.21.15.138:0/2456015639 shutdown_connections 2024-09-11T05:24:24.896 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 -- 172.21.15.138:0/2456015639 wait complete. 2024-09-11T05:24:24.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 10 monclient: build_initial_monmap 2024-09-11T05:24:24.897 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 10 monclient: monmap: 2024-09-11T05:24:24.897 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:24.897 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:24.897 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:24.897 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:24.897 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:24.900 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:24.900 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:24.900 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 librados: starting msgr at 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 librados: starting objecter 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 Processor -- start 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 -- start start 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 librados: setting wanted keys 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 1 librados: calling monclient init 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.890+0000 7f9006300640 10 monclient: init 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9000134dd0 0x7f9000141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:24.901 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient(hunting): picked mon.noname-b con 0x7f9000134dd0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f90001355e0 0x7f900013c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient(hunting): picked mon.noname-c con 0x7f90001355e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9000138a60 0x7f900013c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient(hunting): picked mon.noname-a con 0x7f9000138a60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f90001449c0 con 0x7f9000138a60 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9000144840 con 0x7f9000134dd0 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f9000144b40 con 0x7f90001355e0 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient(hunting): authenticate will time out at 1948.332117s 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9000138a60 0x7f900013c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:24.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9000138a60 0x7f900013c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49312/0 (socket says 172.21.15.138:49312) 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 -- 172.21.15.138:0/3578958015 learned_addr learned my addr 172.21.15.138:0/3578958015 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient(hunting): get_auth_request con 0x7f9000138a60 auth_method 0 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8fff7fe640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f90001355e0 0x7f900013c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8fff7fe640 10 monclient(hunting): get_auth_request con 0x7f90001355e0 auth_method 0 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8fff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8fff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8fff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffffff640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9000134dd0 0x7f9000141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffffff640 10 monclient(hunting): get_auth_request con 0x7f9000134dd0 auth_method 0 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8fff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8fff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8fff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient(hunting): handle_auth_done global_id 114240 payload 824 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 -- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f90001355e0 msgr2=0x7f900013c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f90001355e0 0x7f900013c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 -- 172.21.15.138:0/3578958015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9000134dd0 msgr2=0x7f9000141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:24.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9000134dd0 0x7f9000141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient: _finish_hunting 0 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 monclient: found mon.noname-a 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 -- 172.21.15.138:0/3578958015 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9000140070 con 0x7f9000138a60 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffffff640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9000134dd0 0x7f9000141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 -- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9000138a60 msgr2=0x7f900013c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f900489d640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9000138a60 0x7f900013c7e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800ca40 tx=0x7f8ff800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 client.?.objecter ms_handle_connect 0x7f9000138a60 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 1 -- 172.21.15.138:0/3578958015 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff800ade0 con 0x7f9000138a60 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:24.905 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: dump: 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: _finish_auth 0 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: _check_auth_tickets 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:24.906 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 1 -- 172.21.15.138:0/3578958015 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ff8012400 con 0x7f9000138a60 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 1 -- 172.21.15.138:0/3578958015 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff80127f0 con 0x7f9000138a60 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 5 monclient: authenticate success, global_id 114240 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient: _renew_subs 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90001385c0 con 0x7f9000138a60 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffe7fc640 4 set_mon_vals no callback set 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f8ffcff9640 10 monclient: dump: 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:24.907 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 client.114240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient: _renew_subs 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f900013dd10 con 0x7f9000138a60 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.894+0000 7f9006300640 1 librados: init done 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f9006300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:24.908 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f9006300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f9006300640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90001359c0 con 0x7f9000138a60 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffcff9640 1 -- 172.21.15.138:0/3578958015 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f8ff8012a10 con 0x7f9000138a60 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffcff9640 1 --2- 172.21.15.138:0/3578958015 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8fec03ced0 0x7f8fec03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffcff9640 1 -- 172.21.15.138:0/3578958015 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(691..691 src has 520..691) ==== 5962+0+0 (secure 0 0 0) 0x7f8ff80501d0 con 0x7f9000138a60 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffcff9640 10 client.114240.objecter ms_dispatch 0x7f9000000ba0 osd_map(691..691 src has 520..691) 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffcff9640 3 client.114240.objecter handle_osd_map got epochs [691,691] > 0 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffcff9640 3 client.114240.objecter handle_osd_map decoding full epoch 691 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffcff9640 20 client.114240.objecter dump_active .. 0 homeless 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffffff640 1 --2- 172.21.15.138:0/3578958015 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8fec03ced0 0x7f8fec03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:24.909 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.898+0000 7f8ffffff640 10 monclient: get_auth_request con 0x7f8fec03ced0 auth_method 0 2024-09-11T05:24:24.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.902+0000 7f8ffffff640 1 --2- 172.21.15.138:0/3578958015 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8fec03ced0 0x7f8fec03f390 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00ab000 tx=0x7f8ff0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:24.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.902+0000 7f8ffcff9640 10 client.114240.objecter ms_handle_connect 0x7f8fec03ced0 2024-09-11T05:24:24.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.902+0000 7f8ffcff9640 1 -- 172.21.15.138:0/3578958015 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f8ff8012d30 con 0x7f9000138a60 2024-09-11T05:24:24.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.902+0000 7f8ffcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:24.910 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:24.902+0000 7f8ffcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:24.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.930+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.930+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=2002702479}) 2024-09-11T05:24:24.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.930+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=46} crc {osdmap=1602082278}) 2024-09-11T05:24:24.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.930+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:24:24.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.930+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=2002702479}) 2024-09-11T05:24:24.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:24:24.930+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=45} crc {osdmap=2121512163}) 2024-09-11T05:24:25.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.142+0000 7f9006300640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:24:25.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.142+0000 7f9006300640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:25.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.142+0000 7f9006300640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:25.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.142+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f90000a1920 con 0x7f9000138a60 2024-09-11T05:24:25.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f8ffcff9640 1 -- 172.21.15.138:0/3578958015 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f8ff801bc00 con 0x7f9000138a60 2024-09-11T05:24:25.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f8ffcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:25.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f8ffcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:25.143 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:24:25.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 10 librados: watch_flush enter 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 10 librados: watch_flush exit 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 20 client.114240.objecter shutdown clearing up homeless session... 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 10 client.114240.objecter successfully canceled tick 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8fec03ced0 msgr2=0x7f8fec03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 --2- 172.21.15.138:0/3578958015 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8fec03ced0 0x7f8fec03f390 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00ab000 tx=0x7f8ff0008040 comp rx=0 tx=0).stop 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 10 monclient: shutdown 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9000138a60 msgr2=0x7f900013c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9000138a60 0x7f900013c7e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800ca40 tx=0x7f8ff800cf00 comp rx=0 tx=0).stop 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 shutdown_connections 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 --2- 172.21.15.138:0/3578958015 >> v2:172.21.15.190:6800/1562782189 conn(0x7f8fec03ced0 0x7f8fec03f390 unknown :-1 s=CLOSED pgs=3504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9000138a60 0x7f900013c7e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f90001355e0 0x7f900013c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 --2- 172.21.15.138:0/3578958015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9000134dd0 0x7f9000141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 >> 172.21.15.138:0/3578958015 conn(0x7f9000063640 msgr2=0x7f9000133190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 shutdown_connections 2024-09-11T05:24:25.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 -- 172.21.15.138:0/3578958015 wait complete. 2024-09-11T05:24:25.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.146+0000 7f9006300640 1 librados: shutdown 2024-09-11T05:24:25.162 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:24:25.162 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T05:24:25.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:24:25.163 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:24:25.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient: get_monmap_and_config 2024-09-11T05:24:25.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient: build_initial_monmap 2024-09-11T05:24:25.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient: monmap: 2024-09-11T05:24:25.327 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:25.333801+0000 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:25.333801+0000 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:25.328 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 1 Processor -- start 2024-09-11T05:24:25.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 1 -- start start 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient: init 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a54070010 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a5409e6e0 0x7f1a5409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5409e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1a5413d640 0x7f1a5413fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient(hunting): picked mon.noname-c con 0x7f1a5413d640 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a54135b60 con 0x7f1a54070010 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1a540703f0 con 0x7f1a5413d640 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1a540a8990 con 0x7f1a5409e6e0 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5b2af640 10 monclient(hunting): authenticate will time out at 1948.769900s 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a5409e6e0 0x7f1a5409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a5409e6e0 0x7f1a5409eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:35632/0 (socket says 172.21.15.138:35632) 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 1 -- 172.21.15.138:0/752558665 learned_addr learned my addr 172.21.15.138:0/752558665 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient(hunting): get_auth_request con 0x7f1a5409e6e0 auth_method 0 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5984c640 1 --2- 172.21.15.138:0/752558665 >> v2:172.21.15.190:3300/0 conn(0x7f1a5413d640 0x7f1a5413fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5984c640 10 monclient(hunting): get_auth_request con 0x7f1a5413d640 auth_method 0 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5984c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5984c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5984c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient(hunting): handle_auth_done global_id 84548 payload 274 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 1 -- 172.21.15.138:0/752558665 >> v2:172.21.15.190:3300/0 conn(0x7f1a5413d640 msgr2=0x7f1a5413fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 1 --2- 172.21.15.138:0/752558665 >> v2:172.21.15.190:3300/0 conn(0x7f1a5413d640 0x7f1a5413fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 1 -- 172.21.15.138:0/752558665 >> v1:172.21.15.190:6789/0 conn(0x7f1a54070010 legacy=0x7f1a54135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient: _finish_hunting 0 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 1 monclient: found mon.noname-b 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.330+0000 7f1a5884a640 1 -- 172.21.15.138:0/752558665 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5409cbf0 con 0x7f1a5409e6e0 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5884a640 1 -- 172.21.15.138:0/752558665 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a5409e6e0 msgr2=0x7f1a5409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5884a640 1 --2- 172.21.15.138:0/752558665 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a5409e6e0 0x7f1a5409eae0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009e50 tx=0x7f1a4401c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=828c9b23300a1292 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 1 -- 172.21.15.138:0/752558665 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a440a5640 con 0x7f1a5409e6e0 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: dump: 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.338 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: _finish_auth 0 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: _check_auth_tickets 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 5 monclient: authenticate success, global_id 84548 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 1 -- 172.21.15.138:0/752558665 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a440a57e0 con 0x7f1a5409e6e0 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:25.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 4 set_mon_vals no callback set 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 1 -- 172.21.15.138:0/752558665 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a440a5bb0 con 0x7f1a5409e6e0 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a53fff640 10 monclient: dump: 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.340 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 10 monclient: shutdown 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 1 -- 172.21.15.138:0/752558665 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a5409e6e0 msgr2=0x7f1a5409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 1 --2- 172.21.15.138:0/752558665 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a5409e6e0 0x7f1a5409eae0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009e50 tx=0x7f1a4401c8f0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 1 -- 172.21.15.138:0/752558665 shutdown_connections 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 1 --2- 172.21.15.138:0/752558665 >> v2:172.21.15.190:3300/0 conn(0x7f1a5413d640 0x7f1a5413fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 1 --2- 172.21.15.138:0/752558665 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a5409e6e0 0x7f1a5409eae0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 1 -- 172.21.15.138:0/752558665 >> 172.21.15.138:0/752558665 conn(0x7f1a540ab830 msgr2=0x7f1a540a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.334+0000 7f1a5b2af640 1 -- 172.21.15.138:0/752558665 shutdown_connections 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 -- 172.21.15.138:0/752558665 wait complete. 2024-09-11T05:24:25.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient: build_initial_monmap 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient: monmap: 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 librados: starting msgr at 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 librados: starting objecter 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 Processor -- start 2024-09-11T05:24:25.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 -- start start 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 librados: setting wanted keys 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 librados: calling monclient init 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient: init 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a54070010 0x7f1a54071c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient(hunting): picked mon.noname-c con 0x7f1a54070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409e6e0 0x7f1a540a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient(hunting): picked mon.noname-a con 0x7f1a5409e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5413d640 0x7f1a540a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient(hunting): picked mon.noname-b con 0x7f1a5413d640 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54142bd0 con 0x7f1a5409e6e0 2024-09-11T05:24:25.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54142a50 con 0x7f1a5413d640 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1a54142d50 con 0x7f1a54070010 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5b2af640 10 monclient(hunting): authenticate will time out at 1948.777431s 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a54070010 0x7f1a54071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a54070010 0x7f1a54071c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:35648/0 (socket says 172.21.15.138:35648) 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 -- 172.21.15.138:0/4256174842 learned_addr learned my addr 172.21.15.138:0/4256174842 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient(hunting): get_auth_request con 0x7f1a54070010 auth_method 0 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5884a640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409e6e0 0x7f1a540a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5884a640 10 monclient(hunting): get_auth_request con 0x7f1a5409e6e0 auth_method 0 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5884a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5884a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5884a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5984c640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5413d640 0x7f1a540a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5984c640 10 monclient(hunting): get_auth_request con 0x7f1a5413d640 auth_method 0 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5984c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5984c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5984c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient(hunting): handle_auth_done global_id 84551 payload 824 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5413d640 msgr2=0x7f1a540a3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a5413d640 0x7f1a540a3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409e6e0 msgr2=0x7f1a540a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409e6e0 0x7f1a540a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient: _finish_hunting 0 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 monclient: found mon.noname-c 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.338+0000 7f1a5904b640 1 -- 172.21.15.138:0/4256174842 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5409e0d0 con 0x7f1a54070010 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5904b640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a54070010 msgr2=0x7f1a54071c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5904b640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a54070010 0x7f1a54071c40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00c450 tx=0x7f1a4c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 client.?.objecter ms_handle_connect 0x7f1a54070010 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c012a40 con 0x7f1a54070010 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: dump: 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.351 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: _finish_auth 0 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a4c017070 con 0x7f1a54070010 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 5 monclient: authenticate success, global_id 84551 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 10 monclient: _renew_subs 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a540a2330 con 0x7f1a54070010 2024-09-11T05:24:25.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c012e40 con 0x7f1a54070010 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 monclient: dump: 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.353 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 10 client.84551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 10 monclient: _renew_subs 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a52ffd640 4 set_mon_vals no callback set 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a54072d60 con 0x7f1a54070010 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 1 librados: init done 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a540a4200 0x7f1a5409f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1a540a45e0 con 0x7f1a540a4200 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a540a4200 0x7f1a5409f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 10 monclient: get_auth_request con 0x7f1a540a4200 auth_method 0 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1a4c01c460 con 0x7f1a54070010 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 10 monclient(hunting): handle_auth_done global_id 114246 payload 824 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 1 -- 172.21.15.138:0/4256174842 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a5409eae0 con 0x7f1a540a4200 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 1 --2- 172.21.15.138:0/4256174842 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a4003ced0 0x7f1a4003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(692..692 src has 520..692) ==== 5576+0+0 (secure 0 0 0) 0x7f1a4c04f7f0 con 0x7f1a54070010 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 client.84551.objecter ms_dispatch 0x7f1a54000ba0 osd_map(692..692 src has 520..692) 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 3 client.84551.objecter handle_osd_map got epochs [692,692] > 0 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 3 client.84551.objecter handle_osd_map decoding full epoch 692 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 20 client.84551.objecter dump_active .. 0 homeless 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5984c640 1 --2- 172.21.15.138:0/4256174842 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a4003ced0 0x7f1a4003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5984c640 10 monclient: get_auth_request con 0x7f1a4003ced0 auth_method 0 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a540a4200 msgr2=0x7f1a5409f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 client.84551.objecter ms_handle_connect 0x7f1a540a4200 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a51ffb640 10 client.84551.objecter resend_mon_ops 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.342+0000 7f1a5884a640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a540a4200 0x7f1a5409f030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a440ae000 tx=0x7f1a44019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a440b6800 con 0x7f1a540a4200 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 10 monclient: dump: 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.356 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1a440b6800 con 0x7f1a540a4200 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a540a4200 msgr2=0x7f1a5409f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a540a4200 0x7f1a5409f030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a440ae000 tx=0x7f1a44019690 comp rx=0 tx=0).stop 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a540a4200 msgr2=0x7f1a5409f030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a540a4200 0x7f1a5409f030 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a51ffb640 10 client.84551.objecter ms_handle_connect 0x7f1a4003ced0 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.346+0000 7f1a5984c640 1 --2- 172.21.15.138:0/4256174842 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a4003ced0 0x7f1a4003f390 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7f1a540a9810 tx=0x7f1a48008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5b2af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5b2af640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409f570 0x7f1a5409f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5b2af640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54072910 con 0x7f1a5409f570 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409f570 0x7f1a5409f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 10 monclient: get_auth_request con 0x7f1a5409f570 auth_method 0 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 10 monclient(hunting): handle_auth_done global_id 114249 payload 824 2024-09-11T05:24:25.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.350+0000 7f1a5884a640 1 -- 172.21.15.138:0/4256174842 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1a54072180 con 0x7f1a5409f570 2024-09-11T05:24:25.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5884a640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409f570 msgr2=0x7f1a5409f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 10 client.84551.objecter ms_handle_connect 0x7f1a5409f570 2024-09-11T05:24:25.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 10 client.84551.objecter resend_mon_ops 2024-09-11T05:24:25.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5884a640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409f570 0x7f1a5409f950 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a4401c720 tx=0x7f1a440a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44004040 con 0x7f1a5409f570 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 10 monclient: dump: 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.360 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f1a44003fe0 con 0x7f1a5409f570 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409f570 msgr2=0x7f1a5409f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409f570 0x7f1a5409f950 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a4401c720 tx=0x7f1a440a7000 comp rx=0 tx=0).stop 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409f570 msgr2=0x7f1a5409f950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a51ffb640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1a5409f570 0x7f1a5409f950 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a4401c720 tx=0x7f1a440a7000 comp rx=0 tx=0).stop 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:25.361 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 64, 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 137393, 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 4, 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:24:25.362 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:24:25.363 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.364 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:24:25.365 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.366 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:24:25.367 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.368 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:25.370 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:24:25.371 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.372 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.374 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:24:25.374 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:25.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 10 librados: watch_flush enter 2024-09-11T05:24:25.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 10 librados: watch_flush exit 2024-09-11T05:24:25.375 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 20 client.84551.objecter shutdown clearing up homeless session... 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 10 client.84551.objecter successfully canceled tick 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a4003ced0 msgr2=0x7f1a4003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 --2- 172.21.15.138:0/4256174842 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1a4003ced0 0x7f1a4003f390 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7f1a540a9810 tx=0x7f1a48008040 comp rx=0 tx=0).stop 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 10 monclient: shutdown 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a54070010 msgr2=0x7f1a54071c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 --2- 172.21.15.138:0/4256174842 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1a54070010 0x7f1a54071c40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00c450 tx=0x7f1a4c00c910 comp rx=0 tx=0).stop 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5984c640 1 -- 172.21.15.138:0/4256174842 reap_dead start 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5984c640 1 -- 172.21.15.138:0/4256174842 reap_dead start 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 shutdown_connections 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 >> 172.21.15.138:0/4256174842 conn(0x7f1a540ab830 msgr2=0x7f1a54133260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 shutdown_connections 2024-09-11T05:24:25.376 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 -- 172.21.15.138:0/4256174842 wait complete. 2024-09-11T05:24:25.377 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.354+0000 7f1a5b2af640 1 librados: shutdown 2024-09-11T05:24:25.377 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:24:25.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:24:25.377 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:24:25.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.490+0000 7fb366112640 10 monclient: get_monmap_and_config 2024-09-11T05:24:25.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.490+0000 7fb366112640 10 monclient: build_initial_monmap 2024-09-11T05:24:25.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.490+0000 7fb366112640 10 monclient: monmap: 2024-09-11T05:24:25.490 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:25.496195+0000 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:25.496195+0000 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.490+0000 7fb366112640 1 Processor -- start 2024-09-11T05:24:25.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.490+0000 7fb366112640 1 -- start start 2024-09-11T05:24:25.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.490+0000 7fb366112640 10 monclient: init 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb360070010 0x7fb360135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient(hunting): picked mon.noname-c con 0x7fb360070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb36009e6e0 0x7fb36009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient(hunting): picked mon.noname-b con 0x7fb36009e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fb3601382c0 0x7fb360138770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient(hunting): picked mon.noname-a con 0x7fb3601382c0 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fb3600703f0 con 0x7fb3601382c0 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb3600a8990 con 0x7fb360070010 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb36009cbf0 con 0x7fb36009e6e0 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb366112640 10 monclient(hunting): authenticate will time out at 1948.932221s 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb36009e6e0 0x7fb36009eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35ffff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fb3601382c0 0x7fb360138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb36009e6e0 0x7fb36009eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:35650/0 (socket says 172.21.15.138:35650) 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 -- 172.21.15.138:0/2510409974 learned_addr learned my addr 172.21.15.138:0/2510409974 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient(hunting): get_auth_request con 0x7fb36009e6e0 auth_method 0 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35ffff640 10 monclient(hunting): get_auth_request con 0x7fb3601382c0 auth_method 0 2024-09-11T05:24:25.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35f7fe640 1 --2- 172.21.15.138:0/2510409974 >> v2:172.21.15.190:3300/0 conn(0x7fb360070010 0x7fb360135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35f7fe640 10 monclient(hunting): get_auth_request con 0x7fb360070010 auth_method 0 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient(hunting): handle_auth_done global_id 84554 payload 274 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 -- 172.21.15.138:0/2510409974 >> v2:172.21.15.190:3300/0 conn(0x7fb360070010 msgr2=0x7fb360135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 --2- 172.21.15.138:0/2510409974 >> v2:172.21.15.190:3300/0 conn(0x7fb360070010 0x7fb360135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 -- 172.21.15.138:0/2510409974 >> v2:172.21.15.138:3300/0 conn(0x7fb3601382c0 msgr2=0x7fb360138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 --2- 172.21.15.138:0/2510409974 >> v2:172.21.15.138:3300/0 conn(0x7fb3601382c0 0x7fb360138770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient: _finish_hunting 0 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 monclient: found mon.noname-b 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.494+0000 7fb35effd640 1 -- 172.21.15.138:0/2510409974 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36009ced0 con 0x7fb36009e6e0 2024-09-11T05:24:25.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35effd640 1 -- 172.21.15.138:0/2510409974 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb36009e6e0 msgr2=0x7fb36009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35effd640 1 --2- 172.21.15.138:0/2510409974 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb36009e6e0 0x7fb36009eae0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb36006daa0 tx=0x7fb35401c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efd34fb674e543a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 1 -- 172.21.15.138:0/2510409974 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3540b2070 con 0x7fb36009e6e0 2024-09-11T05:24:25.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: dump: 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.504 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 1 -- 172.21.15.138:0/2510409974 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3540ad640 con 0x7fb36009e6e0 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 1 -- 172.21.15.138:0/2510409974 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3540ada10 con 0x7fb36009e6e0 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 5 monclient: authenticate success, global_id 84554 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 4 set_mon_vals no callback set 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35e7fc640 10 monclient: dump: 2024-09-11T05:24:25.505 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient: shutdown 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- 172.21.15.138:0/2510409974 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb36009e6e0 msgr2=0x7fb36009eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 --2- 172.21.15.138:0/2510409974 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb36009e6e0 0x7fb36009eae0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb36006daa0 tx=0x7fb35401c3b0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- 172.21.15.138:0/2510409974 shutdown_connections 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 --2- 172.21.15.138:0/2510409974 >> v2:172.21.15.138:3300/0 conn(0x7fb3601382c0 0x7fb360138770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 --2- 172.21.15.138:0/2510409974 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb36009e6e0 0x7fb36009eae0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 --2- 172.21.15.138:0/2510409974 >> v2:172.21.15.190:3300/0 conn(0x7fb360070010 0x7fb360135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- 172.21.15.138:0/2510409974 >> 172.21.15.138:0/2510409974 conn(0x7fb3600ab830 msgr2=0x7fb3600a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- 172.21.15.138:0/2510409974 shutdown_connections 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- 172.21.15.138:0/2510409974 wait complete. 2024-09-11T05:24:25.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient: build_initial_monmap 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient: monmap: 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 librados: starting msgr at 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 librados: starting objecter 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 Processor -- start 2024-09-11T05:24:25.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- start start 2024-09-11T05:24:25.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 librados: setting wanted keys 2024-09-11T05:24:25.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 librados: calling monclient init 2024-09-11T05:24:25.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient: init 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb360070010 0x7fb3601d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient(hunting): picked mon.noname-a con 0x7fb360070010 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36009e6e0 0x7fb36013ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient(hunting): picked mon.noname-b con 0x7fb36009e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3601382c0 0x7fb36013ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient(hunting): picked mon.noname-c con 0x7fb3601382c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fb360141e70 con 0x7fb360070010 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb360141ff0 con 0x7fb36009e6e0 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb360142170 con 0x7fb3601382c0 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb366112640 10 monclient(hunting): authenticate will time out at 1948.939616s 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb360070010 0x7fb3601d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3601382c0 0x7fb36013ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb360070010 0x7fb3601d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49352/0 (socket says 172.21.15.138:49352) 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 1 -- 172.21.15.138:0/383607420 learned_addr learned my addr 172.21.15.138:0/383607420 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 10 monclient(hunting): get_auth_request con 0x7fb360070010 auth_method 0 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35ffff640 10 monclient(hunting): get_auth_request con 0x7fb3601382c0 auth_method 0 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.498+0000 7fb35f7fe640 10 monclient(hunting): handle_auth_done global_id 114255 payload 824 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36009e6e0 0x7fb36013ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3601382c0 msgr2=0x7fb36013ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3601382c0 0x7fb36013ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36009e6e0 msgr2=0x7fb36013ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb36009e6e0 0x7fb36013ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 1 monclient: found mon.noname-a 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 1 -- 172.21.15.138:0/383607420 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb360132db0 con 0x7fb360070010 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb360070010 msgr2=0x7fb3601d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35f7fe640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb360070010 0x7fb3601d0690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb35000d800 tx=0x7fb35000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 client.?.objecter ms_handle_connect 0x7fb360070010 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:25.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35000b8d0 con 0x7fb360070010 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: dump: 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: _finish_auth 0 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: _check_auth_tickets 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 5 monclient: authenticate success, global_id 114255 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb350019070 con 0x7fb360070010 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35000bc70 con 0x7fb360070010 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb347fff640 10 monclient: dump: 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.519 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35dffb640 4 set_mon_vals no callback set 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 10 monclient: _renew_subs 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36013b1f0 con 0x7fb360070010 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 10 client.114255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 10 monclient: _renew_subs 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:25.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb360139610 con 0x7fb360070010 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 1 librados: init done 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0590 0x7fb3600a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb360135ba0 con 0x7fb3600a0590 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0590 0x7fb3600a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 10 monclient: get_auth_request con 0x7fb3600a0590 auth_method 0 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 10 monclient(hunting): handle_auth_done global_id 84560 payload 824 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.502+0000 7fb35effd640 1 -- 172.21.15.138:0/383607420 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb36009e140 con 0x7fb3600a0590 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fb350013070 con 0x7fb360070010 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 --2- 172.21.15.138:0/383607420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb34c03cea0 0x7fb34c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(692..692 src has 520..692) ==== 5576+0+0 (secure 0 0 0) 0x7fb35004fc00 con 0x7fb360070010 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 client.114255.objecter ms_dispatch 0x7fb360000ba0 osd_map(692..692 src has 520..692) 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 3 client.114255.objecter handle_osd_map got epochs [692,692] > 0 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 3 client.114255.objecter handle_osd_map decoding full epoch 692 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 20 client.114255.objecter dump_active .. 0 homeless 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb35ffff640 1 --2- 172.21.15.138:0/383607420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb34c03cea0 0x7fb34c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb35ffff640 10 monclient: get_auth_request con 0x7fb34c03cea0 auth_method 0 2024-09-11T05:24:25.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 client.114255.objecter ms_handle_connect 0x7fb3600a0590 2024-09-11T05:24:25.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb35effd640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0590 msgr2=0x7fb3600a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 client.114255.objecter resend_mon_ops 2024-09-11T05:24:25.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb35effd640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0590 0x7fb3600a0970 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb3600a7a00 tx=0x7fb354019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3540b2050 con 0x7fb3600a0590 2024-09-11T05:24:25.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 monclient: dump: 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb3540b3040 con 0x7fb3600a0590 2024-09-11T05:24:25.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0590 msgr2=0x7fb3600a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0590 0x7fb3600a0970 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb3600a7a00 tx=0x7fb354019150 comp rx=0 tx=0).stop 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0590 msgr2=0x7fb3600a0970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0590 0x7fb3600a0970 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb3600a7a00 tx=0x7fb354019150 comp rx=0 tx=0).stop 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb347fff640 10 client.114255.objecter ms_handle_connect 0x7fb34c03cea0 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.506+0000 7fb35ffff640 1 --2- 172.21.15.138:0/383607420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb34c03cea0 0x7fb34c03f360 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7fb360133880 tx=0x7fb35800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb366112640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:24:25.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb366112640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0eb0 0x7fb3600a9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb366112640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fb360135450 con 0x7fb3600a0eb0 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0eb0 0x7fb3600a9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 10 monclient: get_auth_request con 0x7fb3600a0eb0 auth_method 0 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 10 monclient(hunting): handle_auth_done global_id 84563 payload 824 2024-09-11T05:24:25.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 1 -- 172.21.15.138:0/383607420 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb360069fa0 con 0x7fb3600a0eb0 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0eb0 msgr2=0x7fb3600a9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 10 client.114255.objecter ms_handle_connect 0x7fb3600a0eb0 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 10 client.114255.objecter resend_mon_ops 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb35effd640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0eb0 0x7fb3600a9b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb35401cc50 tx=0x7fb3540a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3540b7d10 con 0x7fb3600a0eb0 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 10 monclient: dump: 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.529 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fb3540c8420 con 0x7fb3600a0eb0 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0eb0 msgr2=0x7fb3600a9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0eb0 0x7fb3600a9b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb35401cc50 tx=0x7fb3540a7000 comp rx=0 tx=0).stop 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0eb0 msgr2=0x7fb3600a9b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.514+0000 7fb347fff640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fb3600a0eb0 0x7fb3600a9b10 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb35401cc50 tx=0x7fb3540a7000 comp rx=0 tx=0).stop 2024-09-11T05:24:25.530 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 64, 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 509284, 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 4, 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:24:25.531 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.532 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:24:25.533 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:25.534 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.535 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.538 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.539 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:24:25.540 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:24:25.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.542 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:25.543 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:25.543 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:24:25.543 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.543 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.543 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:24:25.543 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:25.544 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 10 librados: watch_flush enter 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 10 librados: watch_flush exit 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 20 client.114255.objecter shutdown clearing up homeless session... 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 10 client.114255.objecter successfully canceled tick 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb34c03cea0 msgr2=0x7fb34c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 --2- 172.21.15.138:0/383607420 >> v2:172.21.15.190:6800/1562782189 conn(0x7fb34c03cea0 0x7fb34c03f360 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7fb360133880 tx=0x7fb35800b040 comp rx=0 tx=0).stop 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 10 monclient: shutdown 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb360070010 msgr2=0x7fb3601d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.545 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb360070010 0x7fb3601d0690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb35000d800 tx=0x7fb35000dcc0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb35ffff640 1 -- 172.21.15.138:0/383607420 reap_dead start 2024-09-11T05:24:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 shutdown_connections 2024-09-11T05:24:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 --2- 172.21.15.138:0/383607420 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fb360070010 0x7fb3601d0690 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 >> 172.21.15.138:0/383607420 conn(0x7fb3600ab830 msgr2=0x7fb36009d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 shutdown_connections 2024-09-11T05:24:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 -- 172.21.15.138:0/383607420 wait complete. 2024-09-11T05:24:25.546 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.518+0000 7fb366112640 1 librados: shutdown 2024-09-11T05:24:25.546 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:24:25.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:24:25.546 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:24:25.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient: get_monmap_and_config 2024-09-11T05:24:25.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient: build_initial_monmap 2024-09-11T05:24:25.663 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient: monmap: 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:25.669717+0000 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:25.669717+0000 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:25.664 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 1 Processor -- start 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 1 -- start start 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient: init 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a640a32c0 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f6a6409eb30 0x7f6a6413a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient(hunting): picked mon.noname-a con 0x7f6a6409eb30 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409ef50 0x7f6a6413ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient(hunting): picked mon.noname-b con 0x7f6a6409ef50 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a640a1b50 con 0x7f6a640a32c0 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f6a6409fe70 con 0x7f6a6409eb30 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6a640a1920 con 0x7f6a6409ef50 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a69791640 10 monclient(hunting): authenticate will time out at 1949.105737s 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a62ffd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f6a640a32c0 0x7f6a6409e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33302/0 (socket says 172.21.15.138:33302) 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a62ffd640 1 -- 172.21.15.138:0/38461638 learned_addr learned my addr 172.21.15.138:0/38461638 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a627fc640 1 --2- 172.21.15.138:0/38461638 >> v2:172.21.15.138:3300/0 conn(0x7f6a6409eb30 0x7f6a6413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 1 --2- 172.21.15.138:0/38461638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409ef50 0x7f6a6413ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a627fc640 10 monclient(hunting): get_auth_request con 0x7f6a6409eb30 auth_method 0 2024-09-11T05:24:25.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient(hunting): get_auth_request con 0x7f6a6409ef50 auth_method 0 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient(hunting): handle_auth_done global_id 84566 payload 274 2024-09-11T05:24:25.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 1 -- 172.21.15.138:0/38461638 >> v2:172.21.15.138:3300/0 conn(0x7f6a6409eb30 msgr2=0x7f6a6413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:25.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 1 --2- 172.21.15.138:0/38461638 >> v2:172.21.15.138:3300/0 conn(0x7f6a6409eb30 0x7f6a6413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 1 -- 172.21.15.138:0/38461638 >> v1:172.21.15.138:6789/0 conn(0x7f6a640a32c0 legacy=0x7f6a6409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:25.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient: _finish_hunting 0 2024-09-11T05:24:25.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 1 monclient: found mon.noname-b 2024-09-11T05:24:25.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:25.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.666+0000 7f6a637fe640 1 -- 172.21.15.138:0/38461638 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6409f560 con 0x7f6a6409ef50 2024-09-11T05:24:25.673 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a637fe640 1 -- 172.21.15.138:0/38461638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409ef50 msgr2=0x7f6a6413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a637fe640 1 --2- 172.21.15.138:0/38461638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409ef50 0x7f6a6413ce80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0097c0 tx=0x7f6a5c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81cf5f9523dd30c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 1 -- 172.21.15.138:0/38461638 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0b0070 con 0x7f6a6409ef50 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: dump: 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.674 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: _finish_auth 0 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 1 -- 172.21.15.138:0/38461638 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a5c0ab600 con 0x7f6a6409ef50 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 5 monclient: authenticate success, global_id 84566 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:25.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 1 -- 172.21.15.138:0/38461638 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0ab9d0 con 0x7f6a6409ef50 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 4 set_mon_vals no callback set 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a61ffb640 10 monclient: dump: 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.676 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 10 monclient: shutdown 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 1 -- 172.21.15.138:0/38461638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409ef50 msgr2=0x7f6a6413ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 1 --2- 172.21.15.138:0/38461638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409ef50 0x7f6a6413ce80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0097c0 tx=0x7f6a5c01c360 comp rx=0 tx=0).stop 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 1 -- 172.21.15.138:0/38461638 shutdown_connections 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 1 --2- 172.21.15.138:0/38461638 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409ef50 0x7f6a6413ce80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 1 --2- 172.21.15.138:0/38461638 >> v2:172.21.15.138:3300/0 conn(0x7f6a6409eb30 0x7f6a6413a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 1 -- 172.21.15.138:0/38461638 >> 172.21.15.138:0/38461638 conn(0x7f6a64063640 msgr2=0x7f6a64063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 1 -- 172.21.15.138:0/38461638 shutdown_connections 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 1 -- 172.21.15.138:0/38461638 wait complete. 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 10 monclient: build_initial_monmap 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.670+0000 7f6a69791640 10 monclient: monmap: 2024-09-11T05:24:25.677 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 librados: starting msgr at 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 librados: starting objecter 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 Processor -- start 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 -- start start 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 librados: setting wanted keys 2024-09-11T05:24:25.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 librados: calling monclient init 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient: init 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409eb30 0x7f6a641d0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient(hunting): picked mon.noname-c con 0x7f6a6409eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6409ef50 0x7f6a641359f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient(hunting): picked mon.noname-b con 0x7f6a6409ef50 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a640a32c0 0x7f6a64135f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient(hunting): picked mon.noname-a con 0x7f6a640a32c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6413fbe0 con 0x7f6a640a32c0 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6406e280 con 0x7f6a6409ef50 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6413fd60 con 0x7f6a6409eb30 2024-09-11T05:24:25.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient(hunting): authenticate will time out at 1949.112764s 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a640a32c0 0x7f6a64135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409eb30 0x7f6a641d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a640a32c0 0x7f6a64135f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:49378/0 (socket says 172.21.15.138:49378) 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409eb30 0x7f6a641d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:35686/0 (socket says 172.21.15.138:35686) 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 1 -- 172.21.15.138:0/3950310787 learned_addr learned my addr 172.21.15.138:0/3950310787 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 10 monclient(hunting): get_auth_request con 0x7f6a640a32c0 auth_method 0 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient(hunting): get_auth_request con 0x7f6a6409eb30 auth_method 0 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient(hunting): handle_auth_done global_id 84569 payload 824 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a627fc640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6409ef50 0x7f6a641359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6409ef50 msgr2=0x7f6a641359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6409ef50 0x7f6a641359f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a640a32c0 msgr2=0x7f6a64135f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a640a32c0 0x7f6a64135f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:24:25.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 monclient: found mon.noname-c 2024-09-11T05:24:25.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:25.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 -- 172.21.15.138:0/3950310787 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6409e110 con 0x7f6a6409eb30 2024-09-11T05:24:25.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a637fe640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6a640a32c0 0x7f6a64135f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:24:25.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409eb30 msgr2=0x7f6a641d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a62ffd640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409eb30 0x7f6a641d0680 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00caf0 tx=0x7f6a4c00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 client.?.objecter ms_handle_connect 0x7f6a6409eb30 2024-09-11T05:24:25.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:25.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c012400 con 0x7f6a6409eb30 2024-09-11T05:24:25.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:25.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: dump: 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: _finish_auth 0 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:24:25.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a4c0125a0 con 0x7f6a6409eb30 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c012990 con 0x7f6a6409eb30 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 5 monclient: authenticate success, global_id 84569 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient: _renew_subs 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a641233b0 con 0x7f6a6409eb30 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a5b7fe640 10 monclient: dump: 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.687 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a617fa640 4 set_mon_vals no callback set 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 client.84569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient: _renew_subs 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a64122b10 con 0x7f6a6409eb30 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.674+0000 7f6a69791640 1 librados: init done 2024-09-11T05:24:25.688 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a69791640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a69791640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6411fd50 0x7f6a6402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a69791640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6409e420 con 0x7f6a6411fd50 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f6a4c012bf0 con 0x7f6a6409eb30 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6411fd50 0x7f6a6402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 10 monclient: get_auth_request con 0x7f6a6411fd50 auth_method 0 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 1 --2- 172.21.15.138:0/3950310787 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6a4403cef0 0x7f6a4403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.689 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(692..692 src has 520..692) ==== 5576+0+0 (secure 0 0 0) 0x7f6a4c0500f0 con 0x7f6a6409eb30 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 10 client.84569.objecter ms_dispatch 0x7f6a64000ba0 osd_map(692..692 src has 520..692) 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 3 client.84569.objecter handle_osd_map got epochs [692,692] > 0 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 3 client.84569.objecter handle_osd_map decoding full epoch 692 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 20 client.84569.objecter dump_active .. 0 homeless 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a637fe640 1 --2- 172.21.15.138:0/3950310787 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6a4403cef0 0x7f6a4403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a637fe640 10 monclient: get_auth_request con 0x7f6a4403cef0 auth_method 0 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 10 monclient(hunting): handle_auth_done global_id 124138 payload 824 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 1 -- 172.21.15.138:0/3950310787 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a64072a00 con 0x7f6a6411fd50 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6411fd50 msgr2=0x7f6a6402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a627fc640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6411fd50 0x7f6a6402d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a641235c0 tx=0x7f6a54010c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 10 client.84569.objecter ms_handle_connect 0x7f6a6411fd50 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.678+0000 7f6a5b7fe640 10 client.84569.objecter resend_mon_ops 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 10 client.84569.objecter ms_handle_connect 0x7f6a4403cef0 2024-09-11T05:24:25.690 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a637fe640 1 --2- 172.21.15.138:0/3950310787 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6a4403cef0 0x7f6a4403f3b0 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c01c3a0 tx=0x7f6a5c0a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5400f040 con 0x7f6a6411fd50 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 10 monclient: dump: 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.691 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6a5401c400 con 0x7f6a6411fd50 2024-09-11T05:24:25.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:25.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:25.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6411fd50 msgr2=0x7f6a6402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6411fd50 0x7f6a6402d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a641235c0 tx=0x7f6a54010c20 comp rx=0 tx=0).stop 2024-09-11T05:24:25.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6411fd50 msgr2=0x7f6a6402d450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:25.692 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.682+0000 7f6a5b7fe640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a6411fd50 0x7f6a6402d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a69791640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a69791640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a64120130 0x7f6a6402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a69791640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6406cca0 con 0x7f6a64120130 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a64120130 0x7f6a6402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 10 monclient: get_auth_request con 0x7f6a64120130 auth_method 0 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.694 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.686+0000 7f6a627fc640 10 monclient(hunting): handle_auth_done global_id 124141 payload 824 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a627fc640 1 -- 172.21.15.138:0/3950310787 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6a64121bb0 con 0x7f6a64120130 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a627fc640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a64120130 msgr2=0x7f6a6402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a627fc640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a64120130 0x7f6a6402d990 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a641235c0 tx=0x7f6a540046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 10 client.84569.objecter ms_handle_connect 0x7f6a64120130 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 10 client.84569.objecter resend_mon_ops 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5401c8b0 con 0x7f6a64120130 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 10 monclient: dump: 2024-09-11T05:24:25.695 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f6a54025070 con 0x7f6a64120130 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:25.696 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a64120130 msgr2=0x7f6a6402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a64120130 0x7f6a6402d990 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a641235c0 tx=0x7f6a540046d0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a64120130 msgr2=0x7f6a6402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.690+0000 7f6a5b7fe640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a64120130 0x7f6a6402d990 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a641235c0 tx=0x7f6a540046d0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 64, 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 13603, 2024-09-11T05:24:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 4, 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:24:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:24:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:24:25.708 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:25.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 10 librados: watch_flush enter 2024-09-11T05:24:25.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 10 librados: watch_flush exit 2024-09-11T05:24:25.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 20 client.84569.objecter shutdown clearing up homeless session... 2024-09-11T05:24:25.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 10 client.84569.objecter successfully canceled tick 2024-09-11T05:24:25.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6a4403cef0 msgr2=0x7f6a4403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.710 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 --2- 172.21.15.138:0/3950310787 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6a4403cef0 0x7f6a4403f3b0 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c01c3a0 tx=0x7f6a5c0a6000 comp rx=0 tx=0).stop 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 10 monclient: shutdown 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409eb30 msgr2=0x7f6a641d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409eb30 0x7f6a641d0680 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00caf0 tx=0x7f6a4c00a060 comp rx=0 tx=0).stop 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a637fe640 1 -- 172.21.15.138:0/3950310787 reap_dead start 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 shutdown_connections 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 --2- 172.21.15.138:0/3950310787 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6a6409eb30 0x7f6a641d0680 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 >> 172.21.15.138:0/3950310787 conn(0x7f6a64063640 msgr2=0x7f6a6413be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 shutdown_connections 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 -- 172.21.15.138:0/3950310787 wait complete. 2024-09-11T05:24:25.711 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.694+0000 7f6a69791640 1 librados: shutdown 2024-09-11T05:24:25.711 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:24:25.712 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T05:24:25.712 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-11T05:24:25.712 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T05:24:25.712 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-11T05:24:25.712 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-11T05:24:25.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient: get_monmap_and_config 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient: build_initial_monmap 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient: monmap: 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:25.834721+0000 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:25.834721+0000 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.829 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:25.830 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:25.830 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:25.830 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:25.830 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:25.830 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 1 Processor -- start 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 1 -- start start 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient: init 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00e8138a70 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00e8138e50 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f00e8138200 0x7f00e81415b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient(hunting): picked mon.noname-a con 0x7f00e8138200 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00e80a25d0 con 0x7f00e8138a70 2024-09-11T05:24:25.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00e8135b20 con 0x7f00e8138e50 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f00e8138630 con 0x7f00e8138200 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00ed4d2640 10 monclient(hunting): authenticate will time out at 1949.270956s 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f00e8138200 0x7f00e81415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e6ffd640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f00e8138a70 0x7f00e8134d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:33316/0 (socket says 172.21.15.138:33316) 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e6ffd640 1 -- 172.21.15.138:0/4116540730 learned_addr learned my addr 172.21.15.138:0/4116540730 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient(hunting): get_auth_request con 0x7f00e8138200 auth_method 0 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient(hunting): handle_auth_done global_id 114264 payload 274 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 1 -- 172.21.15.138:0/4116540730 >> v1:172.21.15.190:6789/0 conn(0x7f00e8138e50 legacy=0x7f00e8135410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 1 -- 172.21.15.138:0/4116540730 >> v1:172.21.15.138:6789/0 conn(0x7f00e8138a70 legacy=0x7f00e8134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient: _finish_hunting 0 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 1 monclient: found mon.noname-a 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:25.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.830+0000 7f00e77fe640 1 -- 172.21.15.138:0/4116540730 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e80a28b0 con 0x7f00e8138200 2024-09-11T05:24:25.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e77fe640 1 -- 172.21.15.138:0/4116540730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138200 msgr2=0x7f00e81415b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e77fe640 1 --2- 172.21.15.138:0/4116540730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138200 0x7f00e81415b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f00d80097f0 tx=0x7f00d801c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c6192bec2843d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 1 -- 172.21.15.138:0/4116540730 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d80b0070 con 0x7f00e8138200 2024-09-11T05:24:25.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:25.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: dump: 2024-09-11T05:24:25.839 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.839 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: _finish_auth 0 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 1 -- 172.21.15.138:0/4116540730 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d80ab600 con 0x7f00e8138200 2024-09-11T05:24:25.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 1 -- 172.21.15.138:0/4116540730 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d80ab9d0 con 0x7f00e8138200 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00ed4d2640 5 monclient: authenticate success, global_id 114264 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00ed4d2640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00ed4d2640 4 set_mon_vals no callback set 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.834+0000 7f00e5ffb640 10 monclient: dump: 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.841 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient: shutdown 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- 172.21.15.138:0/4116540730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138200 msgr2=0x7f00e81415b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 --2- 172.21.15.138:0/4116540730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138200 0x7f00e81415b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f00d80097f0 tx=0x7f00d801c180 comp rx=0 tx=0).stop 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- 172.21.15.138:0/4116540730 shutdown_connections 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 --2- 172.21.15.138:0/4116540730 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138200 0x7f00e81415b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- 172.21.15.138:0/4116540730 >> 172.21.15.138:0/4116540730 conn(0x7f00e80a3b00 msgr2=0x7f00e80a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- 172.21.15.138:0/4116540730 shutdown_connections 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- 172.21.15.138:0/4116540730 wait complete. 2024-09-11T05:24:25.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient: build_initial_monmap 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient: monmap: 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 librados: starting msgr at 2024-09-11T05:24:25.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 librados: starting objecter 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 Processor -- start 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- start start 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 librados: setting wanted keys 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 librados: calling monclient init 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient: init 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f00e8138200 0x7f00e81319f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient(hunting): picked mon.noname-c con 0x7f00e8138200 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138a70 0x7f00e8131f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient(hunting): picked mon.noname-a con 0x7f00e8138a70 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e8138e50 0x7f00e813b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient(hunting): picked mon.noname-b con 0x7f00e8138e50 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8144720 con 0x7f00e8138a70 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f00e80a38f0 con 0x7f00e8138e50 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f00e81448a0 con 0x7f00e8138200 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00ed4d2640 10 monclient(hunting): authenticate will time out at 1949.278282s 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f00e8138200 0x7f00e81319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f00e8138200 0x7f00e81319f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:35694/0 (socket says 172.21.15.138:35694) 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 -- 172.21.15.138:0/711654671 learned_addr learned my addr 172.21.15.138:0/711654671 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:25.845 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient(hunting): get_auth_request con 0x7f00e8138200 auth_method 0 2024-09-11T05:24:25.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e67fc640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138a70 0x7f00e8131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.846 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e67fc640 10 monclient(hunting): get_auth_request con 0x7f00e8138a70 auth_method 0 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e77fe640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e8138e50 0x7f00e813b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e77fe640 10 monclient(hunting): get_auth_request con 0x7f00e8138e50 auth_method 0 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient(hunting): handle_auth_done global_id 84572 payload 824 2024-09-11T05:24:25.847 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e8138e50 msgr2=0x7f00e813b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e8138e50 0x7f00e813b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138a70 msgr2=0x7f00e8131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f00e8138a70 0x7f00e8131f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:24:25.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 monclient: found mon.noname-c 2024-09-11T05:24:25.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:25.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.848 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.838+0000 7f00e6ffd640 1 -- 172.21.15.138:0/711654671 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e813f7f0 con 0x7f00e8138200 2024-09-11T05:24:25.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e6ffd640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f00e8138200 msgr2=0x7f00e81319f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e6ffd640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f00e8138200 0x7f00e81319f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f00e000cc10 tx=0x7f00e000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 client.?.objecter ms_handle_connect 0x7f00e8138200 2024-09-11T05:24:25.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0012400 con 0x7f00e8138200 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: dump: 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.854 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: _finish_auth 0 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00e00125e0 con 0x7f00e8138200 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e00129d0 con 0x7f00e8138200 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 monclient: dump: 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.855 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 5 monclient: authenticate success, global_id 84572 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 10 monclient: _renew_subs 2024-09-11T05:24:25.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e4ff9640 4 set_mon_vals no callback set 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e8141690 con 0x7f00e8138200 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 10 client.84572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 10 monclient: _renew_subs 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e813edf0 con 0x7f00e8138200 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 1 librados: init done 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e813bdd0 0x7f00e813c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8134d00 con 0x7f00e813bdd0 2024-09-11T05:24:25.857 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f00e0012bf0 con 0x7f00e8138200 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e813bdd0 0x7f00e813c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 10 monclient: get_auth_request con 0x7f00e813bdd0 auth_method 0 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 1 --2- 172.21.15.138:0/711654671 >> v2:172.21.15.190:6800/1562782189 conn(0x7f00cc03cf00 0x7f00cc03f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(692..692 src has 520..692) ==== 5576+0+0 (secure 0 0 0) 0x7f00e00501e0 con 0x7f00e8138200 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 10 client.84572.objecter ms_dispatch 0x7f00e8000ba0 osd_map(692..692 src has 520..692) 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 3 client.84572.objecter handle_osd_map got epochs [692,692] > 0 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 3 client.84572.objecter handle_osd_map decoding full epoch 692 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.858 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00df7fe640 20 client.84572.objecter dump_active .. 0 homeless 2024-09-11T05:24:25.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e77fe640 1 --2- 172.21.15.138:0/711654671 >> v2:172.21.15.190:6800/1562782189 conn(0x7f00cc03cf00 0x7f00cc03f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 10 monclient(hunting): handle_auth_done global_id 124147 payload 824 2024-09-11T05:24:25.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e67fc640 1 -- 172.21.15.138:0/711654671 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00e806e070 con 0x7f00e813bdd0 2024-09-11T05:24:25.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.842+0000 7f00e77fe640 10 monclient: get_auth_request con 0x7f00cc03cf00 auth_method 0 2024-09-11T05:24:25.859 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00e67fc640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e813bdd0 msgr2=0x7f00e813c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00e67fc640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e813bdd0 0x7f00e813c1b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00d0007fd0 tx=0x7f00d000f9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 10 client.84572.objecter ms_handle_connect 0x7f00e813bdd0 2024-09-11T05:24:25.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 10 client.84572.objecter resend_mon_ops 2024-09-11T05:24:25.860 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d0003600 con 0x7f00e813bdd0 2024-09-11T05:24:25.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:24:25.861 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 10 monclient: dump: 2024-09-11T05:24:25.861 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.861 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.861 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 10 client.84572.objecter ms_handle_connect 0x7f00cc03cf00 2024-09-11T05:24:25.862 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00e77fe640 1 --2- 172.21.15.138:0/711654671 >> v2:172.21.15.190:6800/1562782189 conn(0x7f00cc03cf00 0x7f00cc03f3c0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f00e806de90 tx=0x7f00d80a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f00d0018070 con 0x7f00e813bdd0 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e813bdd0 msgr2=0x7f00e813c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e813bdd0 0x7f00e813c1b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00d0007fd0 tx=0x7f00d000f9f0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e813bdd0 msgr2=0x7f00e813c1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.846+0000 7f00df7fe640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e813bdd0 0x7f00e813c1b0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00d0007fd0 tx=0x7f00d000f9f0 comp rx=0 tx=0).stop 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00ed4d2640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T05:24:25.863 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00ed4d2640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e802d450 0x7f00e802d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:25.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00ed4d2640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:25.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f00e809fb10 con 0x7f00e802d450 2024-09-11T05:24:25.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e802d450 0x7f00e802d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:25.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 10 monclient: get_auth_request con 0x7f00e802d450 auth_method 0 2024-09-11T05:24:25.864 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:25.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:25.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:25.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:25.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:25.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:25.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 10 monclient(hunting): handle_auth_done global_id 124150 payload 824 2024-09-11T05:24:25.865 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.854+0000 7f00e67fc640 1 -- 172.21.15.138:0/711654671 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f00e806cd10 con 0x7f00e802d450 2024-09-11T05:24:25.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.858+0000 7f00e67fc640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e802d450 msgr2=0x7f00e802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:25.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.858+0000 7f00df7fe640 10 client.84572.objecter ms_handle_connect 0x7f00e802d450 2024-09-11T05:24:25.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.858+0000 7f00df7fe640 10 client.84572.objecter resend_mon_ops 2024-09-11T05:24:25.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.858+0000 7f00e67fc640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e802d450 0x7f00e802d830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f00d0007fd0 tx=0x7f00d0001030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:25.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.858+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d002d080 con 0x7f00e802d450 2024-09-11T05:24:25.866 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.858+0000 7f00df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:25.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.858+0000 7f00df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:24:25.867 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.858+0000 7f00df7fe640 10 monclient: dump: 2024-09-11T05:24:25.867 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:25.867 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:25.867 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.867 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:25.867 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:25.867 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:25.868 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:25.868 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:25.868 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:25.868 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:25.868 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f00d0003cf0 con 0x7f00e802d450 2024-09-11T05:24:25.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00df7fe640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T05:24:25.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:25.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e802d450 msgr2=0x7f00e802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00df7fe640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e802d450 0x7f00e802d830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f00d0007fd0 tx=0x7f00d0001030 comp rx=0 tx=0).stop 2024-09-11T05:24:25.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00df7fe640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e802d450 msgr2=0x7f00e802d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:25.869 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00df7fe640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f00e802d450 0x7f00e802d830 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f00d0007fd0 tx=0x7f00d0001030 comp rx=0 tx=0).stop 2024-09-11T05:24:25.870 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:25.870 INFO:teuthology.orchestra.run.smithi138.stdout: "ret": 0, 2024-09-11T05:24:25.870 INFO:teuthology.orchestra.run.smithi138.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T05:24:25.870 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:25.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 10 librados: watch_flush enter 2024-09-11T05:24:25.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 10 librados: watch_flush exit 2024-09-11T05:24:25.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 20 client.84572.objecter shutdown clearing up homeless session... 2024-09-11T05:24:25.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 10 client.84572.objecter successfully canceled tick 2024-09-11T05:24:25.871 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 >> v2:172.21.15.190:6800/1562782189 conn(0x7f00cc03cf00 msgr2=0x7f00cc03f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 --2- 172.21.15.138:0/711654671 >> v2:172.21.15.190:6800/1562782189 conn(0x7f00cc03cf00 0x7f00cc03f3c0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f00e806de90 tx=0x7f00d80a6000 comp rx=0 tx=0).stop 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 10 monclient: shutdown 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f00e8138200 msgr2=0x7f00e81319f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 --2- 172.21.15.138:0/711654671 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f00e8138200 0x7f00e81319f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f00e000cc10 tx=0x7f00e000a430 comp rx=0 tx=0).stop 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00e77fe640 1 -- 172.21.15.138:0/711654671 reap_dead start 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00e77fe640 1 -- 172.21.15.138:0/711654671 reap_dead start 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 shutdown_connections 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 >> 172.21.15.138:0/711654671 conn(0x7f00e80a3b00 msgr2=0x7f00e8133f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 shutdown_connections 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 -- 172.21.15.138:0/711654671 wait complete. 2024-09-11T05:24:25.872 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:25.862+0000 7f00ed4d2640 1 librados: shutdown 2024-09-11T05:24:25.876 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T05:24:25.876 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T05:24:25.876 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:24:25.898 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T05:24:25.898 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T05:24:25.899 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T05:24:25.899 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T05:24:40.900 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T05:24:40.900 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T05:24:40.900 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-11T05:24:40.901 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient: get_monmap_and_config 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient: build_initial_monmap 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient: monmap: 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:41.035271+0000 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:41.035271+0000 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:41.029 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:41.030 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:41.030 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:41.030 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:41.030 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:41.030 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:41.030 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 1 Processor -- start 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 1 -- start start 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient: init 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd406e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bd4136170 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 0x7f1bd40a2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:41.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd4136c60 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bd40a3270 con 0x7f1bd4136170 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd4137070 con 0x7f1bd406e0e0 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd406e4c0 con 0x7f1bd4136c60 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd9487640 10 monclient(hunting): authenticate will time out at 1964.471688s 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd2ffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd27fc640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f1bd4136170 0x7f1bd40a1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34778/0 (socket says 172.21.15.138:34778) 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd27fc640 1 -- 172.21.15.138:0/954688473 learned_addr learned my addr 172.21.15.138:0/954688473 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bd406e0e0 auth_method 0 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:41.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 msgr2=0x7f1bd40a2030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:41.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.029+0000 7f1bd37fe640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 0x7f1bd40a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:41.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.033+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:24:41.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.033+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:41.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.033+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:41.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.033+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:41.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.033+0000 7f1bd27fc640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.138:6789/0 conn(0x7f1bd4136170 legacy=0x7f1bd40a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:41.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.233+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 msgr2=0x7f1bd40a2030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:41.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.233+0000 7f1bd37fe640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 0x7f1bd40a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bd406e0e0 auth_method 0 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:41.236 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.237+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:24:41.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.637+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 msgr2=0x7f1bd40a2030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:41.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.637+0000 7f1bd37fe640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 0x7f1bd40a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:24:41.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:41.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bd406e0e0 auth_method 0 2024-09-11T05:24:41.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:41.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:41.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:41.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:24:41.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:41.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:41.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:41.641+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:24:42.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.441+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 msgr2=0x7f1bd40a2030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:42.436 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.441+0000 7f1bd37fe640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 0x7f1bd40a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:24:42.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:42.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bd406e0e0 auth_method 0 2024-09-11T05:24:42.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:42.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:42.564 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:42.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:24:42.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:42.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:42.565 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:42.445+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:24:44.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 10 monclient(hunting): tick 2024-09-11T05:24:44.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:24:44.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 monclient(hunting): continuing hunt 2024-09-11T05:24:44.031 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 msgr2=0x7f1bd40a2030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.190:3300/0 conn(0x7f1bd4136c60 0x7f1bd40a2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 msgr2=0x7f1bd4136720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bd406e0e0 0x7f1bd4136720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.138:6789/0 conn(0x7f1bd4136170 legacy=0x7f1bd40a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 10 monclient: _add_conns ranks=[2,1] 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb4003770 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bb40062f0 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb400a000 con 0x7f1bb40062f0 2024-09-11T05:24:44.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd4137070 con 0x7f1bb4003770 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd0ff9640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd27fc640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.190:6789/0 conn(0x7f1bb40062f0 legacy=0x7f1bb4008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bb4003770 auth_method 0 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:44.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.033+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:44.231 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd27fc640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.190:6789/0 conn(0x7f1bb40062f0 legacy=0x7f1bb4008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:24:44.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:44.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bb4003770 auth_method 0 2024-09-11T05:24:44.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:44.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:44.232 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:44.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:24:44.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:44.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:44.233 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.237+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bb4003770 auth_method 0 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd27fc640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.190:6789/0 conn(0x7f1bb40062f0 legacy=0x7f1bb4008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:44.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:44.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:44.641+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T05:24:45.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd27fc640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.190:6789/0 conn(0x7f1bb40062f0 legacy=0x7f1bb4008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bb4003770 auth_method 0 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:45.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:45.445+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T05:24:47.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient(hunting): tick 2024-09-11T05:24:47.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T05:24:47.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 1 monclient(hunting): continuing hunt 2024-09-11T05:24:47.032 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 msgr2=0x7f1bb4005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb4003770 0x7f1bb4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.190:6789/0 conn(0x7f1bb40062f0 legacy=0x7f1bb4008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 reap_dead start 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bd4136170 addr v1:172.21.15.138:6789/0 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bb400a910 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bb4005bc0 0x7f1bb4009670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f1bb4005bc0 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb400b2b0 con 0x7f1bd4136170 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb400a000 con 0x7f1bb400a910 2024-09-11T05:24:47.033 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 1 -- 172.21.15.138:0/954688473 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd4137070 con 0x7f1bb4005bc0 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd0ff9640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd27fc640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.190:6789/0 conn(0x7f1bb400a910 legacy=0x7f1bb4008f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 1 --2- 172.21.15.138:0/954688473 >> v2:172.21.15.138:3300/0 conn(0x7f1bb4005bc0 0x7f1bb4009670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient(hunting): get_auth_request con 0x7f1bb4005bc0 auth_method 0 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient(hunting): handle_auth_done global_id 114270 payload 274 2024-09-11T05:24:47.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.190:6789/0 conn(0x7f1bb400a910 legacy=0x7f1bb4008f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:47.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 >> v1:172.21.15.138:6789/0 conn(0x7f1bd4136170 legacy=0x7f1bb400a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:24:47.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:24:47.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 1 monclient: found mon.noname-a 2024-09-11T05:24:47.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:47.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.033+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb400b010 con 0x7f1bb4005bc0 2024-09-11T05:24:47.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd37fe640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bb4005bc0 msgr2=0x7f1bb4009670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd37fe640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bb4005bc0 0x7f1bb4009670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4064580 tx=0x7f1bcc01c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da244e1aa279d50e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd37fe640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bb4005bc0 0x7f1bb4009670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4064580 tx=0x7f1bcc01c7e0 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f1bcc0a86f0 mon_map magic: 0 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 1 -- 172.21.15.138:0/954688473 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc0a86f0 con 0x7f1bb4005bc0 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 10 monclient: dump: 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.040 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 10 monclient: _finish_auth 0 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 1 -- 172.21.15.138:0/954688473 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bcc0ad070 con 0x7f1bb4005bc0 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 5 monclient: authenticate success, global_id 114270 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:47.041 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 4 set_mon_vals no callback set 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 10 monclient: shutdown 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 1 -- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bb4005bc0 msgr2=0x7f1bb4009670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bb4005bc0 0x7f1bb4009670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4064580 tx=0x7f1bcc01c7e0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 1 -- 172.21.15.138:0/954688473 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f1bcc0a8ad0 con 0x7f1bb4005bc0 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd1ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 1 -- 172.21.15.138:0/954688473 shutdown_connections 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 1 --2- 172.21.15.138:0/954688473 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bb4005bc0 0x7f1bb4009670 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 1 -- 172.21.15.138:0/954688473 >> 172.21.15.138:0/954688473 conn(0x7f1bd4063640 msgr2=0x7f1bd4063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:47.043 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 1 -- 172.21.15.138:0/954688473 shutdown_connections 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 1 -- 172.21.15.138:0/954688473 wait complete. 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 10 monclient: build_initial_monmap 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.041+0000 7f1bd9487640 10 monclient: monmap: 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 librados: starting msgr at 2024-09-11T05:24:47.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 librados: starting objecter 2024-09-11T05:24:47.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 Processor -- start 2024-09-11T05:24:47.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 -- start start 2024-09-11T05:24:47.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 librados: setting wanted keys 2024-09-11T05:24:47.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 librados: calling monclient init 2024-09-11T05:24:47.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient: init 2024-09-11T05:24:47.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:47.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:24:47.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb4005bc0 0x7f1bd41ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb4005bc0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:47.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb400a910 0x7f1bd411e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient(hunting): picked mon.noname-c con 0x7f1bb400a910 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bd4136170 0x7f1bd411e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd4136170 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd40a24c0 con 0x7f1bd4136170 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd40a2340 con 0x7f1bb4005bc0 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd40a2640 con 0x7f1bb400a910 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd9487640 10 monclient(hunting): authenticate will time out at 1970.485331s 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd2ffd640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb4005bc0 msgr2=0x7f1bd41ceb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd2ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb4005bc0 0x7f1bd41ceb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb400a910 0x7f1bd411e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bd4136170 0x7f1bd411e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb400a910 0x7f1bd411e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:57006/0 (socket says 172.21.15.138:57006) 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 -- 172.21.15.138:0/3699232934 learned_addr learned my addr 172.21.15.138:0/3699232934 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient(hunting): get_auth_request con 0x7f1bb400a910 auth_method 0 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 10 monclient(hunting): get_auth_request con 0x7f1bd4136170 auth_method 0 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient(hunting): handle_auth_done global_id 84575 payload 824 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 -- 172.21.15.138:0/3699232934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb4005bc0 msgr2=0x7f1bd41ceb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 --2- 172.21.15.138:0/3699232934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb4005bc0 0x7f1bd41ceb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 -- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bd4136170 msgr2=0x7f1bd411e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 --2- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bd4136170 0x7f1bd411e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient: _finish_hunting 0 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 monclient: found mon.noname-c 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd27fc640 1 -- 172.21.15.138:0/3699232934 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd411fce0 con 0x7f1bb400a910 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.045+0000 7f1bd37fe640 1 --2- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bd4136170 0x7f1bd411e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:24:47.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd27fc640 1 -- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb400a910 msgr2=0x7f1bd411e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd27fc640 1 --2- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb400a910 0x7f1bd411e220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00c730 tx=0x7f1bbc00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 client.?.objecter ms_handle_connect 0x7f1bb400a910 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 1 -- 172.21.15.138:0/3699232934 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc017070 con 0x7f1bb400a910 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: dump: 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.053 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: _finish_auth 0 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 1 -- 172.21.15.138:0/3699232934 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bbc005cf0 con 0x7f1bb400a910 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 5 monclient: authenticate success, global_id 84575 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 10 monclient: _renew_subs 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd409e110 con 0x7f1bb400a910 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 1 -- 172.21.15.138:0/3699232934 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc007270 con 0x7f1bb400a910 2024-09-11T05:24:47.054 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 monclient: dump: 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd17fa640 4 set_mon_vals no callback set 2024-09-11T05:24:47.055 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 10 client.84575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 10 monclient: _renew_subs 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd41217a0 con 0x7f1bb400a910 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 1 librados: init done 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 1 -- 172.21.15.138:0/3699232934 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1bbc0074d0 con 0x7f1bb400a910 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd4139550 con 0x7f1bb400a910 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 1 --2- 172.21.15.138:0/3699232934 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1bb403e930 0x7f1bb4040df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 1 -- 172.21.15.138:0/3699232934 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(694..694 src has 520..694) ==== 5962+0+0 (secure 0 0 0) 0x7f1bbc04f430 con 0x7f1bb400a910 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 10 client.84575.objecter ms_dispatch 0x7f1bd4000ba0 osd_map(694..694 src has 520..694) 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 3 client.84575.objecter handle_osd_map got epochs [694,694] > 0 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 3 client.84575.objecter handle_osd_map decoding full epoch 694 2024-09-11T05:24:47.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bcb7fe640 20 client.84575.objecter dump_active .. 0 homeless 2024-09-11T05:24:47.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/3699232934 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1bb403e930 0x7f1bb4040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.049+0000 7f1bd2ffd640 10 monclient: get_auth_request con 0x7f1bb403e930 auth_method 0 2024-09-11T05:24:47.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.053+0000 7f1bd2ffd640 1 --2- 172.21.15.138:0/3699232934 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1bb403e930 0x7f1bb4040df0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40094c0 tx=0x7f1bc400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.053+0000 7f1bcb7fe640 10 client.84575.objecter ms_handle_connect 0x7f1bb403e930 2024-09-11T05:24:47.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.057+0000 7f1bcb7fe640 1 -- 172.21.15.138:0/3699232934 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f1bd40a3270 con 0x7f1bb400a910 2024-09-11T05:24:47.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.057+0000 7f1bcb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:47.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.057+0000 7f1bcb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:47.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:24:47.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:47.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:24:47.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1bd406ccd0 con 0x7f1bb400a910 2024-09-11T05:24:47.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bcb7fe640 1 -- 172.21.15.138:0/3699232934 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f1bbc04d990 con 0x7f1bb400a910 2024-09-11T05:24:47.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bcb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:24:47.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bcb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:47.301 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":66,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 10 librados: watch_flush enter 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 10 librados: watch_flush exit 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 20 client.84575.objecter shutdown clearing up homeless session... 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 10 client.84575.objecter successfully canceled tick 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1bb403e930 msgr2=0x7f1bb4040df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 --2- 172.21.15.138:0/3699232934 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1bb403e930 0x7f1bb4040df0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40094c0 tx=0x7f1bc400b040 comp rx=0 tx=0).stop 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 10 monclient: shutdown 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb400a910 msgr2=0x7f1bd411e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 --2- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb400a910 0x7f1bd411e220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00c730 tx=0x7f1bbc00cbf0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 shutdown_connections 2024-09-11T05:24:47.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 --2- 172.21.15.138:0/3699232934 >> v2:172.21.15.190:6800/1562782189 conn(0x7f1bb403e930 0x7f1bb4040df0 unknown :-1 s=CLOSED pgs=3509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 --2- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1bd4136170 0x7f1bd411e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 --2- 172.21.15.138:0/3699232934 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1bb400a910 0x7f1bd411e220 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 --2- 172.21.15.138:0/3699232934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb4005bc0 0x7f1bd41ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.301+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 >> 172.21.15.138:0/3699232934 conn(0x7f1bd4063640 msgr2=0x7f1bd40a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:47.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.305+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 shutdown_connections 2024-09-11T05:24:47.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.305+0000 7f1bd9487640 1 -- 172.21.15.138:0/3699232934 wait complete. 2024-09-11T05:24:47.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.305+0000 7f1bd9487640 1 librados: shutdown 2024-09-11T05:24:47.319 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-11T05:24:47.320 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-11T05:24:47.320 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:24:47.439 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient: get_monmap_and_config 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient: build_initial_monmap 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient: monmap: 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:47.446234+0000 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:47.446234+0000 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:47.440 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 1 Processor -- start 2024-09-11T05:24:47.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 1 -- start start 2024-09-11T05:24:47.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient: init 2024-09-11T05:24:47.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:47.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-11T05:24:47.443 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe94009da20 0x7fe9400a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient(hunting): picked mon.noname-c con 0x7fe94009da20 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe9400a0a90 0x7fe9400a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient(hunting): picked mon.noname-a con 0x7fe9400a0a90 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe940134850 addr v1:172.21.15.190:6789/0 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9400a13e0 con 0x7fe940134850 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fe94009efa0 con 0x7fe9400a0a90 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe94009de00 con 0x7fe94009da20 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe944f3d640 10 monclient(hunting): authenticate will time out at 1970.883066s 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe93e59c640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7fe94009da20 msgr2=0x7fe9400a26c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:47.444 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe93e59c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe94009da20 0x7fe9400a26c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe93ed9d640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7fe940134850 legacy=0x7fe940134c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe9400a0a90 0x7fe9400a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fe9400a0a90 0x7fe9400a0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43152/0 (socket says 172.21.15.138:43152) 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 1 -- 172.21.15.138:0/2904853041 learned_addr learned my addr 172.21.15.138:0/2904853041 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient(hunting): get_auth_request con 0x7fe9400a0a90 auth_method 0 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient(hunting): handle_auth_done global_id 114276 payload 274 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 1 -- 172.21.15.138:0/2904853041 >> v2:172.21.15.190:3300/0 conn(0x7fe94009da20 msgr2=0x7fe9400a26c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:47.445 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 1 --2- 172.21.15.138:0/2904853041 >> v2:172.21.15.190:3300/0 conn(0x7fe94009da20 0x7fe9400a26c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 1 -- 172.21.15.138:0/2904853041 >> v1:172.21.15.190:6789/0 conn(0x7fe940134850 legacy=0x7fe940134c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:47.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:24:47.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 1 monclient: found mon.noname-a 2024-09-11T05:24:47.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:47.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.446 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.441+0000 7fe935d9b640 1 -- 172.21.15.138:0/2904853041 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe940064590 con 0x7fe9400a0a90 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.445+0000 7fe935d9b640 1 -- 172.21.15.138:0/2904853041 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 msgr2=0x7fe9400a0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.445+0000 7fe935d9b640 1 --2- 172.21.15.138:0/2904853041 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 0x7fe9400a0ea0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe9400a6ef0 tx=0x7fe92801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efea540942c0ef7a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 1 -- 172.21.15.138:0/2904853041 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9280b1070 con 0x7fe9400a0a90 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: dump: 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.447 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: _finish_auth 0 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: _check_auth_tickets 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 1 -- 172.21.15.138:0/2904853041 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9280ac640 con 0x7fe9400a0a90 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 5 monclient: authenticate success, global_id 114276 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 1 -- 172.21.15.138:0/2904853041 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9280aca10 con 0x7fe9400a0a90 2024-09-11T05:24:47.448 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 4 set_mon_vals no callback set 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93dd9b640 10 monclient: dump: 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.449 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient: shutdown 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- 172.21.15.138:0/2904853041 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 msgr2=0x7fe9400a0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 --2- 172.21.15.138:0/2904853041 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 0x7fe9400a0ea0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe9400a6ef0 tx=0x7fe92801c3d0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- 172.21.15.138:0/2904853041 shutdown_connections 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 --2- 172.21.15.138:0/2904853041 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 0x7fe9400a0ea0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 --2- 172.21.15.138:0/2904853041 >> v2:172.21.15.190:3300/0 conn(0x7fe94009da20 0x7fe9400a26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- 172.21.15.138:0/2904853041 >> 172.21.15.138:0/2904853041 conn(0x7fe940063640 msgr2=0x7fe940063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- 172.21.15.138:0/2904853041 shutdown_connections 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- 172.21.15.138:0/2904853041 wait complete. 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient: build_initial_monmap 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient: monmap: 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:47.459 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 librados: starting msgr at 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 librados: starting objecter 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 Processor -- start 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- start start 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 librados: setting wanted keys 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 librados: calling monclient init 2024-09-11T05:24:47.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient: init 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe94009da20 0x7fe940133600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient(hunting): picked mon.noname-c con 0x7fe94009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 0x7fe940133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient(hunting): picked mon.noname-a con 0x7fe9400a0a90 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe940134850 0x7fe94013a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient(hunting): picked mon.noname-b con 0x7fe940134850 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe94013cb90 con 0x7fe9400a0a90 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe94013cd10 con 0x7fe940134850 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fe94013ce90 con 0x7fe94009da20 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe944f3d640 10 monclient(hunting): authenticate will time out at 1970.890423s 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe94009da20 0x7fe940133600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 0x7fe940133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe94009da20 0x7fe940133600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:57008/0 (socket says 172.21.15.138:57008) 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 0x7fe940133b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:43166/0 (socket says 172.21.15.138:43166) 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 1 -- 172.21.15.138:0/3796018133 learned_addr learned my addr 172.21.15.138:0/3796018133 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 10 monclient(hunting): get_auth_request con 0x7fe94009da20 auth_method 0 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient(hunting): get_auth_request con 0x7fe9400a0a90 auth_method 0 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93ed9d640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe940134850 msgr2=0x7fe94013a7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93ed9d640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe940134850 0x7fe94013a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient(hunting): handle_auth_done global_id 114279 payload 824 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe94009da20 msgr2=0x7fe940133600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fe94009da20 0x7fe940133600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe940134850 msgr2=0x7fe94013a7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:24:47.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe940134850 0x7fe94013a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 1 monclient: found mon.noname-a 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.449+0000 7fe935d9b640 1 -- 172.21.15.138:0/3796018133 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9400a07b0 con 0x7fe9400a0a90 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe935d9b640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 msgr2=0x7fe940133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe935d9b640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 0x7fe940133b40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe9280ab000 tx=0x7fe928019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 client.?.objecter ms_handle_connect 0x7fe9400a0a90 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9280b1040 con 0x7fe9400a0a90 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: dump: 2024-09-11T05:24:47.464 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: _finish_auth 0 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: _check_auth_tickets 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 5 monclient: authenticate success, global_id 114279 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9280aa550 con 0x7fe9400a0a90 2024-09-11T05:24:47.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9280aa900 con 0x7fe9400a0a90 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 monclient: dump: 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.466 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93d59a640 4 set_mon_vals no callback set 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 10 monclient: _renew_subs 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9401343f0 con 0x7fe9400a0a90 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 10 client.114279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 10 monclient: _renew_subs 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe94011ed10 con 0x7fe9400a0a90 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 1 librados: init done 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940136920 0x7fe940136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe9400a2e10 con 0x7fe940136920 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940136920 0x7fe940136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 10 monclient: get_auth_request con 0x7fe940136920 auth_method 0 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fe9280aaaa0 con 0x7fe9400a0a90 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 10 monclient(hunting): handle_auth_done global_id 114282 payload 824 2024-09-11T05:24:47.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93e59c640 1 -- 172.21.15.138:0/3796018133 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9400a2740 con 0x7fe940136920 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 1 --2- 172.21.15.138:0/3796018133 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe92003cea0 0x7fe92003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(694..694 src has 520..694) ==== 5962+0+0 (secure 0 0 0) 0x7fe9280e9330 con 0x7fe9400a0a90 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 10 client.114279.objecter ms_dispatch 0x7fe940000ba0 osd_map(694..694 src has 520..694) 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 3 client.114279.objecter handle_osd_map got epochs [694,694] > 0 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 3 client.114279.objecter handle_osd_map decoding full epoch 694 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe9377fe640 20 client.114279.objecter dump_active .. 0 homeless 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93ed9d640 1 --2- 172.21.15.138:0/3796018133 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe92003cea0 0x7fe92003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.453+0000 7fe93ed9d640 10 monclient: get_auth_request con 0x7fe92003cea0 auth_method 0 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe93e59c640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940136920 msgr2=0x7fe940136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 10 client.114279.objecter ms_handle_connect 0x7fe940136920 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 10 client.114279.objecter resend_mon_ops 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe93e59c640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940136920 0x7fe940136d00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe930009710 tx=0x7fe930009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe930016070 con 0x7fe940136920 2024-09-11T05:24:47.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 10 monclient: dump: 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe930009e40 con 0x7fe940136920 2024-09-11T05:24:47.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:47.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:47.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940136920 msgr2=0x7fe940136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940136920 0x7fe940136d00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe930009710 tx=0x7fe930009b10 comp rx=0 tx=0).stop 2024-09-11T05:24:47.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940136920 msgr2=0x7fe940136d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:47.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940136920 0x7fe940136d00 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe930009710 tx=0x7fe930009b10 comp rx=0 tx=0).stop 2024-09-11T05:24:47.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe9377fe640 10 client.114279.objecter ms_handle_connect 0x7fe92003cea0 2024-09-11T05:24:47.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.457+0000 7fe93ed9d640 1 --2- 172.21.15.138:0/3796018133 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe92003cea0 0x7fe92003f360 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7fe938003cc0 tx=0x7fe938009270 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.461+0000 7fe944f3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.461+0000 7fe944f3d640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940137240 0x7fe94013ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.461+0000 7fe944f3d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.461+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fe94006cca0 con 0x7fe940137240 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940137240 0x7fe94013ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 10 monclient: get_auth_request con 0x7fe940137240 auth_method 0 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 10 monclient(hunting): handle_auth_done global_id 114285 payload 824 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 1 -- 172.21.15.138:0/3796018133 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe94011d7f0 con 0x7fe940137240 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940137240 msgr2=0x7fe94013ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 10 client.114279.objecter ms_handle_connect 0x7fe940137240 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 10 client.114279.objecter resend_mon_ops 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93e59c640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940137240 0x7fe94013ad00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe930009b50 tx=0x7fe9300059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe930016040 con 0x7fe940137240 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 10 monclient: dump: 2024-09-11T05:24:47.474 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fe93000edd0 con 0x7fe940137240 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940137240 msgr2=0x7fe94013ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940137240 0x7fe94013ad00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe930009b50 tx=0x7fe9300059e0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940137240 msgr2=0x7fe94013ad00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe9377fe640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe940137240 0x7fe94013ad00 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe930009b50 tx=0x7fe9300059e0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 66, 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 159505, 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:24:47.476 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:47.477 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:47.478 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:47.479 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:47.480 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:47.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:47.482 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.483 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:47.484 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.485 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.486 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.487 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:24:47.488 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:47.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 10 librados: watch_flush enter 2024-09-11T05:24:47.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 10 librados: watch_flush exit 2024-09-11T05:24:47.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 20 client.114279.objecter shutdown clearing up homeless session... 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 10 client.114279.objecter successfully canceled tick 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe92003cea0 msgr2=0x7fe92003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 --2- 172.21.15.138:0/3796018133 >> v2:172.21.15.190:6800/1562782189 conn(0x7fe92003cea0 0x7fe92003f360 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7fe938003cc0 tx=0x7fe938009270 comp rx=0 tx=0).stop 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 10 monclient: shutdown 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 msgr2=0x7fe940133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 --2- 172.21.15.138:0/3796018133 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fe9400a0a90 0x7fe940133b40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe9280ab000 tx=0x7fe928019170 comp rx=0 tx=0).stop 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93ed9d640 1 -- 172.21.15.138:0/3796018133 reap_dead start 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe93ed9d640 1 -- 172.21.15.138:0/3796018133 reap_dead start 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 shutdown_connections 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 >> 172.21.15.138:0/3796018133 conn(0x7fe940063640 msgr2=0x7fe9400a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 shutdown_connections 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 -- 172.21.15.138:0/3796018133 wait complete. 2024-09-11T05:24:47.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.465+0000 7fe944f3d640 1 librados: shutdown 2024-09-11T05:24:47.491 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:24:47.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-11T05:24:47.491 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient: get_monmap_and_config 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient: build_initial_monmap 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient: monmap: 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:24:47.610783+0000 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:24:47.610783+0000 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:47.604 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 1 Processor -- start 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 1 -- start start 2024-09-11T05:24:47.605 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient: init 2024-09-11T05:24:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:47.608 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-11T05:24:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 0x7f07d0135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient(hunting): picked mon.noname-b con 0x7f07d0070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f07d009e6e0 0x7f07d009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient(hunting): picked mon.noname-c con 0x7f07d009e6e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:24:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f07d01382c0 0x7f07d0138770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.609 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient(hunting): picked mon.noname-a con 0x7f07d01382c0 addr v2:172.21.15.138:3300/0 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f07d00703f0 con 0x7f07d01382c0 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f07d00a8990 con 0x7f07d009e6e0 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f07d009cbf0 con 0x7f07d0070010 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d6e5b640 10 monclient(hunting): authenticate will time out at 1971.046798s 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 0x7f07d0135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07cffff640 1 -- >> v2:172.21.15.190:3300/0 conn(0x7f07d009e6e0 msgr2=0x7f07d009eae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:47.610 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07cffff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f07d009e6e0 0x7f07d009eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 0x7f07d0135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:40772/0 (socket says 172.21.15.138:40772) 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 1 -- 172.21.15.138:0/3930166268 learned_addr learned my addr 172.21.15.138:0/3930166268 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 10 monclient(hunting): get_auth_request con 0x7f07d0070010 auth_method 0 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 1 --2- 172.21.15.138:0/3930166268 >> v2:172.21.15.138:3300/0 conn(0x7f07d01382c0 0x7f07d0138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07cf7fe640 1 -- 172.21.15.138:0/3930166268 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient(hunting): get_auth_request con 0x7f07d01382c0 auth_method 0 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 1 -- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 msgr2=0x7f07d0135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T05:24:47.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 1 -- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 msgr2=0x7f07d0135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 1 --2- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 0x7f07d0135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d4bf7640 1 --2- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 0x7f07d0135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient(hunting): handle_auth_done global_id 114288 payload 274 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 1 -- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 msgr2=0x7f07d0135450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 1 --2- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 0x7f07d0135450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 1 -- 172.21.15.138:0/3930166268 >> v2:172.21.15.190:3300/0 conn(0x7f07d009e6e0 msgr2=0x7f07d009eae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 1 --2- 172.21.15.138:0/3930166268 >> v2:172.21.15.190:3300/0 conn(0x7f07d009e6e0 0x7f07d009eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient: _finish_hunting 0 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 1 monclient: found mon.noname-a 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.612 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.605+0000 7f07d53f8640 1 -- 172.21.15.138:0/3930166268 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d009ced0 con 0x7f07d01382c0 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 1 -- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d01382c0 msgr2=0x7f07d0138770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 1 --2- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d01382c0 0x7f07d0138770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f07d00705b0 tx=0x7f07c801c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd822b2323b71677 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 1 -- 172.21.15.138:0/3930166268 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c80b2070 con 0x7f07d01382c0 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: dump: 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.621 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: _finish_auth 0 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 5 monclient: authenticate success, global_id 114288 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 1 -- 172.21.15.138:0/3930166268 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c80a5880 con 0x7f07d01382c0 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient: get_monmap_and_config success 2024-09-11T05:24:47.622 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 4 set_mon_vals no callback set 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 1 -- 172.21.15.138:0/3930166268 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c80a5c50 con 0x7f07d01382c0 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cf7fe640 10 monclient: dump: 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.623 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient: shutdown 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d01382c0 msgr2=0x7f07d0138770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d01382c0 0x7f07d0138770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f07d00705b0 tx=0x7f07c801c7e0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- 172.21.15.138:0/3930166268 shutdown_connections 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d01382c0 0x7f07d0138770 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/3930166268 >> v2:172.21.15.190:3300/0 conn(0x7f07d009e6e0 0x7f07d009eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/3930166268 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d0070010 0x7f07d0135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- 172.21.15.138:0/3930166268 >> 172.21.15.138:0/3930166268 conn(0x7f07d00ab830 msgr2=0x7f07d00a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- 172.21.15.138:0/3930166268 shutdown_connections 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- 172.21.15.138:0/3930166268 wait complete. 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient: build_initial_monmap 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient: monmap: 2024-09-11T05:24:47.624 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 librados: starting msgr at 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 librados: starting objecter 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 Processor -- start 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- start start 2024-09-11T05:24:47.625 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 librados: setting wanted keys 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 librados: calling monclient init 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient: init 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient: _reopen_session rank -1 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07d0070010 0x7f07d01d0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient(hunting): picked mon.noname-b con 0x7f07d0070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d009e6e0 0x7f07d013e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient(hunting): picked mon.noname-a con 0x7f07d009e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d01382c0 0x7f07d013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient(hunting): picked mon.noname-c con 0x7f07d01382c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0141e70 con 0x7f07d009e6e0 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0141ff0 con 0x7f07d0070010 2024-09-11T05:24:47.626 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f07d0142170 con 0x7f07d01382c0 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient(hunting): _renew_subs 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d6e5b640 10 monclient(hunting): authenticate will time out at 1971.054725s 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d009e6e0 0x7f07d013e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d4bf7640 1 -- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07d0070010 msgr2=0x7f07d01d0770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3300/0 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d009e6e0 0x7f07d013e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:57564/0 (socket says 172.21.15.138:57564) 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d4bf7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07d0070010 0x7f07d01d0770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d01382c0 0x7f07d013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 -- 172.21.15.138:0/311229317 learned_addr learned my addr 172.21.15.138:0/311229317 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient(hunting): get_auth_request con 0x7f07d009e6e0 auth_method 0 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.627 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 10 monclient(hunting): get_auth_request con 0x7f07d01382c0 auth_method 0 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient(hunting): handle_auth_done global_id 114291 payload 824 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d01382c0 msgr2=0x7f07d013edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d01382c0 0x7f07d013edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07d0070010 msgr2=0x7f07d01d0770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T05:24:47.628 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07d0070010 0x7f07d01d0770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient: _finish_hunting 0 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 monclient: found mon.noname-a 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07cffff640 1 -- 172.21.15.138:0/311229317 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d006fa00 con 0x7f07d009e6e0 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.613+0000 7f07d53f8640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d01382c0 0x7f07d013edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cffff640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d009e6e0 msgr2=0x7f07d013e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cffff640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d009e6e0 0x7f07d013e870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f07c400b550 tx=0x7f07c400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 client.?.objecter ms_handle_connect 0x7f07d009e6e0 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4013020 con 0x7f07d009e6e0 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:24:47.629 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: dump: 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: _finish_auth 0 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:24:47.630 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c4005070 con 0x7f07d009e6e0 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 5 monclient: authenticate success, global_id 114291 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 10 monclient: _renew_subs 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d00a1fd0 con 0x7f07d009e6e0 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4005480 con 0x7f07d009e6e0 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 monclient: dump: 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.631 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07ceffd640 4 set_mon_vals no callback set 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 10 client.114291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 10 monclient: _renew_subs 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d0139b20 con 0x7f07d009e6e0 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 1 librados: init done 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0510 0x7f07d00a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.632 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f07d0135ba0 con 0x7f07d00a0510 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0510 0x7f07d00a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 10 monclient: get_auth_request con 0x7f07d00a0510 auth_method 0 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f07c4005620 con 0x7f07d009e6e0 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 10 monclient(hunting): handle_auth_done global_id 84584 payload 824 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d4bf7640 1 -- 172.21.15.138:0/311229317 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07d0132d60 con 0x7f07d00a0510 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 1 --2- 172.21.15.138:0/311229317 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07bc03cea0 0x7f07bc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(694..694 src has 520..694) ==== 5962+0+0 (secure 0 0 0) 0x7f07c404f7d0 con 0x7f07d009e6e0 2024-09-11T05:24:47.633 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 10 client.114291.objecter ms_dispatch 0x7f07d0000ba0 osd_map(694..694 src has 520..694) 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 3 client.114291.objecter handle_osd_map got epochs [694,694] > 0 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 3 client.114291.objecter handle_osd_map decoding full epoch 694 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07cd7fa640 20 client.114291.objecter dump_active .. 0 homeless 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d53f8640 1 --2- 172.21.15.138:0/311229317 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07bc03cea0 0x7f07bc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.617+0000 7f07d53f8640 10 monclient: get_auth_request con 0x7f07bc03cea0 auth_method 0 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 10 client.114291.objecter ms_handle_connect 0x7f07d00a0510 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 10 client.114291.objecter resend_mon_ops 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07d4bf7640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0510 msgr2=0x7f07d00a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07d4bf7640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0510 0x7f07d00a08f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e710 tx=0x7f07c000ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c0017070 con 0x7f07d00a0510 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 10 monclient: dump: 2024-09-11T05:24:47.634 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f07c0004220 con 0x7f07d00a0510 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0510 msgr2=0x7f07d00a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0510 0x7f07d00a08f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e710 tx=0x7f07c000ebd0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.635 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0510 msgr2=0x7f07d00a08f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:47.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0510 0x7f07d00a08f0 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e710 tx=0x7f07c000ebd0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07cd7fa640 10 client.114291.objecter ms_handle_connect 0x7f07bc03cea0 2024-09-11T05:24:47.636 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.621+0000 7f07d53f8640 1 --2- 172.21.15.138:0/311229317 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07bc03cea0 0x7f07bc03f360 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f07d00a7a00 tx=0x7f07c80a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d6e5b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:24:47.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0e30 0x7f07d00a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:24:47.637 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d6e5b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f07d00a2330 con 0x7f07d00a0e30 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d4bf7640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0e30 0x7f07d00a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d4bf7640 10 monclient: get_auth_request con 0x7f07d00a0e30 auth_method 0 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d4bf7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d4bf7640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d4bf7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d4bf7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d4bf7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.625+0000 7f07d4bf7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d4bf7640 10 monclient(hunting): handle_auth_done global_id 84587 payload 824 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d4bf7640 1 -- 172.21.15.138:0/311229317 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07d0135450 con 0x7f07d00a0e30 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d4bf7640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0e30 msgr2=0x7f07d00a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:24:47.638 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d4bf7640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0e30 0x7f07d00a2540 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e710 tx=0x7f07c0004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 10 client.114291.objecter ms_handle_connect 0x7f07d00a0e30 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 10 client.114291.objecter resend_mon_ops 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c000f040 con 0x7f07d00a0e30 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 10 monclient: dump: 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:24:47.639 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f07c0013410 con 0x7f07d00a0e30 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0e30 msgr2=0x7f07d00a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0e30 0x7f07d00a2540 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e710 tx=0x7f07c0004740 comp rx=0 tx=0).stop 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0e30 msgr2=0x7f07d00a2540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07cd7fa640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f07d00a0e30 0x7f07d00a2540 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e710 tx=0x7f07c0004740 comp rx=0 tx=0).stop 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:24:47.640 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 66, 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 531399, 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 1, 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:24:47.641 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:47.642 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:24:47.643 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:24:47.644 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:24:47.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:24:47.646 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:47.647 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:24:47.648 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:24:47.651 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 4 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:24:47.652 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 10 librados: watch_flush enter 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 10 librados: watch_flush exit 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 20 client.114291.objecter shutdown clearing up homeless session... 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 10 client.114291.objecter successfully canceled tick 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07bc03cea0 msgr2=0x7f07bc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/311229317 >> v2:172.21.15.190:6800/1562782189 conn(0x7f07bc03cea0 0x7f07bc03f360 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f07d00a7a00 tx=0x7f07c80a7070 comp rx=0 tx=0).stop 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 10 monclient: shutdown 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:24:47.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d009e6e0 msgr2=0x7f07d013e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:24:47.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d009e6e0 0x7f07d013e870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f07c400b550 tx=0x7f07c400ba10 comp rx=0 tx=0).stop 2024-09-11T05:24:47.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d53f8640 1 -- 172.21.15.138:0/311229317 reap_dead start 2024-09-11T05:24:47.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 shutdown_connections 2024-09-11T05:24:47.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 --2- 172.21.15.138:0/311229317 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f07d009e6e0 0x7f07d013e870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:24:47.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 >> 172.21.15.138:0/311229317 conn(0x7f07d00ab830 msgr2=0x7f07d00a8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:24:47.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 shutdown_connections 2024-09-11T05:24:47.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 -- 172.21.15.138:0/311229317 wait complete. 2024-09-11T05:24:47.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:24:47.629+0000 7f07d6e5b640 1 librados: shutdown 2024-09-11T05:24:47.655 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-11T05:24:47.655 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T05:24:50.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7620 ms) 2024-09-11T05:24:50.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-11T05:24:50.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12797 ms) 2024-09-11T05:24:50.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-11T05:24:50.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7196 ms) 2024-09-11T05:24:50.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-11T05:24:50.050 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12995 ms) 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (22065 ms) 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8081 ms) 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8458 ms) 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,0 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: first is 1726032187 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,0 2024-09-11T05:24:50.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,1726032192,1726032193,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,1726032192,1726032193,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,1726032192,1726032193,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,1726032192,1726032193,1726032195,1726032196,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,1726032192,1726032193,1726032195,1726032196,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032187,1726032189,1726032190,1726032192,1726032193,1726032195,1726032196,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726032190,1726032192,1726032193,1726032195,1726032196,1726032198,1726032199,0 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: first now 1726032190, trimmed 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20771 ms) 2024-09-11T05:24:50.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: foo0 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13730 ms) 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18187 ms) 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (21800 ms) 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5631 ms) 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3028 ms) 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (2353 ms) 2024-09-11T05:24:50.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-11T05:24:50.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (22951 ms) 2024-09-11T05:24:50.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (283916 ms total) 2024-09-11T05:24:50.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 2024-09-11T05:24:50.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-11T05:24:50.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (964378 ms total) 2024-09-11T05:24:50.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-11T05:24:50.055 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28068 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28068 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28242 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28242 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28608 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28608 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28442 2024-09-11T05:24:50.056 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28442 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28306 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28306 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28114 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28114 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28353 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28353 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28772 2024-09-11T05:24:50.057 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28772 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28803 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28803 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28149 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28149 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28632 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28632 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28052 2024-09-11T05:24:50.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28052 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28084 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28084 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28561 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28561 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28044 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28044 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28060 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28060 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28927 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28927 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28276 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28276 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28961 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28961 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28323 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28323 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28705 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28705 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.065 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29014 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 29014 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28297 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28297 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28577 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28577 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28910 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28910 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28658 2024-09-11T05:24:50.066 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28658 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28937 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28937 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28519 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28519 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28224 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28224 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28990 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28990 2024-09-11T05:24:50.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28469 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28469 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28036 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28036 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28076 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28076 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28428 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28428 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28092 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28092 2024-09-11T05:24:50.068 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28502 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28502 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28511 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28511 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28830 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28830 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29044 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 29044 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ exit 0 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ cleanup 2024-09-11T05:24:50.069 INFO:tasks.workunit.client.0.smithi190.stderr:+ pkill -P 28030 2024-09-11T05:24:50.071 INFO:tasks.workunit.client.0.smithi190.stderr:+ true 2024-09-11T05:24:50.071 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T05:24:50.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T05:24:50.085 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-09-11T05:24:50.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T05:24:50.444 DEBUG:teuthology.parallel:result is None 2024-09-11T05:24:50.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T05:24:50.457 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T05:24:50.457 DEBUG:teuthology.orchestra.run.smithi190:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T05:24:50.512 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T05:24:50.513 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-11T05:24:50.522 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-11T05:26:17.656 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T05:26:17.657 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T05:26:17.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T05:26:17.660 INFO:tasks.ceph.mon.b:Started 2024-09-11T05:26:17.661 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T05:26:17.661 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T05:26:17.661 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T05:26:17.760 INFO:tasks.ceph.mon.b.smithi190.stderr:2024-09-11T05:26:17.762+0000 7f8a0a69fe00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 716 -> 0 (sync_force? mon rebuild?) 2024-09-11T05:26:32.662 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T05:26:32.663 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T05:26:32.663 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:26:32.663 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:26:32.792 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient: get_monmap_and_config 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient: build_initial_monmap 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient: monmap: 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:32.799527+0000 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:32.799527+0000 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:32.793 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:32.794 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:32.794 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:32.794 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:32.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 1 Processor -- start 2024-09-11T05:26:32.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 1 -- start start 2024-09-11T05:26:32.797 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient: init 2024-09-11T05:26:32.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:32.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-11T05:26:32.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f97480a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:32.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient(hunting): picked mon.noname-b con 0x7f97480a49c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:32.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f974806fd10 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:32.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97480a52e0 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:32.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f974806ef80 con 0x7f974806fd10 2024-09-11T05:26:32.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97480704a0 con 0x7f97480a52e0 2024-09-11T05:26:32.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:32.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f97480a34e0 con 0x7f97480a49c0 2024-09-11T05:26:32.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:32.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f97505e9640 10 monclient(hunting): authenticate will time out at 2076.235787s 2024-09-11T05:26:32.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f97480a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:32.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f97480a4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55506/0 (socket says 172.21.15.138:55506) 2024-09-11T05:26:32.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 1 -- 172.21.15.138:0/3775388830 learned_addr learned my addr 172.21.15.138:0/3775388830 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:32.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 10 monclient(hunting): get_auth_request con 0x7f97480a49c0 auth_method 0 2024-09-11T05:26:32.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:32.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:32.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:32.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:32.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:32.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.795+0000 7f974e385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:32.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.799+0000 7f974e385640 10 monclient(hunting): handle_auth_done global_id 84590 payload 274 2024-09-11T05:26:32.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.799+0000 7f974e385640 1 -- 172.21.15.138:0/3775388830 >> v1:172.21.15.190:6789/0 conn(0x7f97480a52e0 legacy=0x7f97481402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:32.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.799+0000 7f974e385640 1 -- 172.21.15.138:0/3775388830 >> v1:172.21.15.138:6789/0 conn(0x7f974806fd10 legacy=0x7f974813cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:32.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.799+0000 7f974e385640 10 monclient: _finish_hunting 0 2024-09-11T05:26:32.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.799+0000 7f974e385640 1 monclient: found mon.noname-b 2024-09-11T05:26:32.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.799+0000 7f974e385640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:32.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.799+0000 7f974e385640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:26:32.802 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.799+0000 7f974e385640 1 -- 172.21.15.138:0/3775388830 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f974806f490 con 0x7f97480a49c0 2024-09-11T05:26:32.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974e385640 1 -- 172.21.15.138:0/3775388830 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 msgr2=0x7f97480a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:32.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974e385640 1 --2- 172.21.15.138:0/3775388830 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f97480a4da0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f97440098a0 tx=0x7f974401c2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=142ca979afa2c19d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:32.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 1 -- 172.21.15.138:0/3775388830 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440b0070 con 0x7f97480a49c0 2024-09-11T05:26:32.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:32.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:26:32.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: dump: 2024-09-11T05:26:32.813 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:32.814 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:32.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: _finish_auth 0 2024-09-11T05:26:32.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: _check_auth_tickets 2024-09-11T05:26:32.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:32.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 1 -- 172.21.15.138:0/3775388830 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97440ab600 con 0x7f97480a49c0 2024-09-11T05:26:32.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 5 monclient: authenticate success, global_id 84590 2024-09-11T05:26:32.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:32.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:32.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:32.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 1 -- 172.21.15.138:0/3775388830 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440ab9d0 con 0x7f97480a49c0 2024-09-11T05:26:32.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 4 set_mon_vals no callback set 2024-09-11T05:26:32.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:32.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:32.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974d383640 10 monclient: dump: 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:32.817 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient: shutdown 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- 172.21.15.138:0/3775388830 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 msgr2=0x7f97480a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 --2- 172.21.15.138:0/3775388830 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f97480a4da0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f97440098a0 tx=0x7f974401c2f0 comp rx=0 tx=0).stop 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- 172.21.15.138:0/3775388830 shutdown_connections 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 --2- 172.21.15.138:0/3775388830 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f97480a4da0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- 172.21.15.138:0/3775388830 >> 172.21.15.138:0/3775388830 conn(0x7f97480a9610 msgr2=0x7f97480a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- 172.21.15.138:0/3775388830 shutdown_connections 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- 172.21.15.138:0/3775388830 wait complete. 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient: build_initial_monmap 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient: monmap: 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:32.818 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 librados: starting msgr at 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 librados: starting objecter 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 Processor -- start 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- start start 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 librados: setting wanted keys 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 librados: calling monclient init 2024-09-11T05:26:32.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient: init 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f974806fd10 0x7f9748128ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient(hunting): picked mon.noname-b con 0x7f974806fd10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f9748140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient(hunting): picked mon.noname-c con 0x7f97480a49c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97480a52e0 0x7f9748126590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient(hunting): picked mon.noname-a con 0x7f97480a52e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9748144780 con 0x7f97480a52e0 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:32.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9748144600 con 0x7f974806fd10 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f9748144900 con 0x7f97480a49c0 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f97505e9640 10 monclient(hunting): authenticate will time out at 2076.243542s 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f9748140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f9748140930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55514/0 (socket says 172.21.15.138:55514) 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 1 -- 172.21.15.138:0/2751481131 learned_addr learned my addr 172.21.15.138:0/2751481131 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 10 monclient(hunting): get_auth_request con 0x7f97480a49c0 auth_method 0 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974eb86640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97480a52e0 0x7f9748126590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:32.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974eb86640 10 monclient(hunting): get_auth_request con 0x7f97480a52e0 auth_method 0 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974eb86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974eb86640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974eb86640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974eb86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974eb86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974eb86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974e385640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f974806fd10 0x7f9748128ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974e385640 10 monclient(hunting): get_auth_request con 0x7f974806fd10 auth_method 0 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974e385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974e385640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:32.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974e385640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.803+0000 7f974db84640 10 monclient(hunting): handle_auth_done global_id 84593 payload 824 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 1 -- 172.21.15.138:0/2751481131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f974806fd10 msgr2=0x7f9748128ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f974806fd10 0x7f9748128ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 1 -- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97480a52e0 msgr2=0x7f9748126590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97480a52e0 0x7f9748126590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 10 monclient: _finish_hunting 0 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 1 monclient: found mon.noname-c 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 1 -- 172.21.15.138:0/2751481131 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9748129440 con 0x7f97480a49c0 2024-09-11T05:26:32.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 1 -- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 msgr2=0x7f9748140930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974db84640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f9748140930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f973800c480 tx=0x7f973800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 client.?.objecter ms_handle_connect 0x7f97480a49c0 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 1 -- 172.21.15.138:0/2751481131 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738012a40 con 0x7f97480a49c0 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: dump: 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:32.824 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: _finish_auth 0 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: _check_auth_tickets 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 5 monclient: authenticate success, global_id 84593 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 10 monclient: _renew_subs 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 1 -- 172.21.15.138:0/2751481131 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9738017070 con 0x7f97480a49c0 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97481d17c0 con 0x7f97480a49c0 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 1 -- 172.21.15.138:0/2751481131 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738012e40 con 0x7f97480a49c0 2024-09-11T05:26:32.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 monclient: dump: 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:32.826 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f974cb82640 4 set_mon_vals no callback set 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 10 client.84593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 10 monclient: _renew_subs 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97480a01a0 con 0x7f97480a49c0 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 1 librados: init done 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 1 -- 172.21.15.138:0/2751481131 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f973801c460 con 0x7f97480a49c0 2024-09-11T05:26:32.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:32.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:32.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:32.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97480a4da0 con 0x7f97480a49c0 2024-09-11T05:26:32.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 1 --2- 172.21.15.138:0/2751481131 >> v2:172.21.15.190:6800/1562782189 conn(0x7f972c03cea0 0x7f972c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:32.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 1 -- 172.21.15.138:0/2751481131 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f973804f500 con 0x7f97480a49c0 2024-09-11T05:26:32.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 10 client.84593.objecter ms_dispatch 0x7f9748000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 3 client.84593.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 3 client.84593.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.807+0000 7f973effd640 20 client.84593.objecter dump_active .. 0 homeless 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.811+0000 7f974e385640 1 --2- 172.21.15.138:0/2751481131 >> v2:172.21.15.190:6800/1562782189 conn(0x7f972c03cea0 0x7f972c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.811+0000 7f974e385640 10 monclient: get_auth_request con 0x7f972c03cea0 auth_method 0 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.811+0000 7f973effd640 10 client.84593.objecter ms_handle_connect 0x7f972c03cea0 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.811+0000 7f974e385640 1 --2- 172.21.15.138:0/2751481131 >> v2:172.21.15.190:6800/1562782189 conn(0x7f972c03cea0 0x7f972c03f360 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f97480a6800 tx=0x7f97440a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.815+0000 7f973effd640 1 -- 172.21.15.138:0/2751481131 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f97481d17c0 con 0x7f97480a49c0 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.815+0000 7f973effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:32.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:32.815+0000 7f973effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:33.034 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.035+0000 7f97505e9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:26:33.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.035+0000 7f97505e9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:26:33.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.035+0000 7f97505e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.035+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9748129af0 con 0x7f97480a49c0 2024-09-11T05:26:33.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f973effd640 1 -- 172.21.15.138:0/2751481131 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f973801ac60 con 0x7f97480a49c0 2024-09-11T05:26:33.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f973effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:26:33.035 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f973effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:26:33.036 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:26:33.037 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 10 librados: watch_flush enter 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 10 librados: watch_flush exit 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 20 client.84593.objecter shutdown clearing up homeless session... 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 10 client.84593.objecter successfully canceled tick 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 >> v2:172.21.15.190:6800/1562782189 conn(0x7f972c03cea0 msgr2=0x7f972c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 --2- 172.21.15.138:0/2751481131 >> v2:172.21.15.190:6800/1562782189 conn(0x7f972c03cea0 0x7f972c03f360 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f97480a6800 tx=0x7f97440a6000 comp rx=0 tx=0).stop 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 10 monclient: shutdown 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 msgr2=0x7f9748140930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f9748140930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f973800c480 tx=0x7f973800c940 comp rx=0 tx=0).stop 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 shutdown_connections 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 --2- 172.21.15.138:0/2751481131 >> v2:172.21.15.190:6800/1562782189 conn(0x7f972c03cea0 0x7f972c03f360 unknown :-1 s=CLOSED pgs=3513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97480a52e0 0x7f9748126590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.038 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97480a49c0 0x7f9748140930 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 --2- 172.21.15.138:0/2751481131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f974806fd10 0x7f9748128ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 >> 172.21.15.138:0/2751481131 conn(0x7f97480a9610 msgr2=0x7f974809f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 shutdown_connections 2024-09-11T05:26:33.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 -- 172.21.15.138:0/2751481131 wait complete. 2024-09-11T05:26:33.039 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.039+0000 7f97505e9640 1 librados: shutdown 2024-09-11T05:26:33.052 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:26:33.052 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-11T05:26:33.052 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T05:26:33.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient: get_monmap_and_config 2024-09-11T05:26:33.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient: build_initial_monmap 2024-09-11T05:26:33.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient: monmap: 2024-09-11T05:26:33.159 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:33.159 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.159 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:33.166061+0000 2024-09-11T05:26:33.159 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:33.166061+0000 2024-09-11T05:26:33.160 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:33.160 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.160 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:33.160 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:33.160 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:33.160 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:33.160 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:33.160 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 1 Processor -- start 2024-09-11T05:26:33.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 1 -- start start 2024-09-11T05:26:33.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient: init 2024-09-11T05:26:33.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:33.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-11T05:26:33.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d7809da20 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:33.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d780a2030 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5d780a24a0 0x7f5d7813ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5d780a24a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d78064590 con 0x7f5d7809da20 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d780a2c70 con 0x7f5d780a2030 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f5d7809de00 con 0x7f5d780a24a0 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d7dbcf640 10 monclient(hunting): authenticate will time out at 2076.602375s 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f5d780a24a0 0x7f5d7813ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d777fe640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f5d7809da20 0x7f5d780a1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:49984/0 (socket says 172.21.15.138:49984) 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d777fe640 1 -- 172.21.15.138:0/2207182843 learned_addr learned my addr 172.21.15.138:0/2207182843 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient(hunting): get_auth_request con 0x7f5d780a24a0 auth_method 0 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.163 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient(hunting): handle_auth_done global_id 114297 payload 274 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 1 -- 172.21.15.138:0/2207182843 >> v1:172.21.15.190:6789/0 conn(0x7f5d780a2030 legacy=0x7f5d7813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 1 -- 172.21.15.138:0/2207182843 >> v1:172.21.15.138:6789/0 conn(0x7f5d7809da20 legacy=0x7f5d780a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient: _finish_hunting 0 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 1 monclient: found mon.noname-a 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:33.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.163+0000 7f5d77fff640 1 -- 172.21.15.138:0/2207182843 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d780a2f50 con 0x7f5d780a24a0 2024-09-11T05:26:33.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d77fff640 1 -- 172.21.15.138:0/2207182843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a24a0 msgr2=0x7f5d7813ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d77fff640 1 --2- 172.21.15.138:0/2207182843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a24a0 0x7f5d7813ec50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009d90 tx=0x7f5d7001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa1f45872dd4b6c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 1 -- 172.21.15.138:0/2207182843 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d700b2070 con 0x7f5d780a24a0 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: dump: 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.170 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.171 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.171 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.171 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: _finish_auth 0 2024-09-11T05:26:33.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: _check_auth_tickets 2024-09-11T05:26:33.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:33.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 1 -- 172.21.15.138:0/2207182843 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d700a5970 con 0x7f5d780a24a0 2024-09-11T05:26:33.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d7dbcf640 5 monclient: authenticate success, global_id 114297 2024-09-11T05:26:33.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d7dbcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:33.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:33.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d7dbcf640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:33.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 1 -- 172.21.15.138:0/2207182843 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d700a5d40 con 0x7f5d780a24a0 2024-09-11T05:26:33.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d7dbcf640 4 set_mon_vals no callback set 2024-09-11T05:26:33.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:33.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d767fc640 10 monclient: dump: 2024-09-11T05:26:33.173 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.173 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.173 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.174 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.174 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.174 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.174 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.174 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.174 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.174 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d7dbcf640 10 monclient: shutdown 2024-09-11T05:26:33.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d7dbcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/2207182843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a24a0 msgr2=0x7f5d7813ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.167+0000 7f5d7dbcf640 1 --2- 172.21.15.138:0/2207182843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a24a0 0x7f5d7813ec50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009d90 tx=0x7f5d7001c830 comp rx=0 tx=0).stop 2024-09-11T05:26:33.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/2207182843 shutdown_connections 2024-09-11T05:26:33.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 --2- 172.21.15.138:0/2207182843 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a24a0 0x7f5d7813ec50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/2207182843 >> 172.21.15.138:0/2207182843 conn(0x7f5d78063640 msgr2=0x7f5d78063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/2207182843 shutdown_connections 2024-09-11T05:26:33.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/2207182843 wait complete. 2024-09-11T05:26:33.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient: build_initial_monmap 2024-09-11T05:26:33.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient: monmap: 2024-09-11T05:26:33.176 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:33.176 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:33.176 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:33.177 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:33.177 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:33.177 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.177 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:33.177 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:33.177 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:33.178 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 librados: starting msgr at 2024-09-11T05:26:33.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 librados: starting objecter 2024-09-11T05:26:33.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 Processor -- start 2024-09-11T05:26:33.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 -- start start 2024-09-11T05:26:33.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 librados: setting wanted keys 2024-09-11T05:26:33.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 librados: calling monclient init 2024-09-11T05:26:33.178 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient: init 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d7809da20 0x7f5d7813cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient(hunting): picked mon.noname-b con 0x7f5d7809da20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a2030 0x7f5d78070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5d780a2030 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5d780a24a0 0x7f5d78070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient(hunting): picked mon.noname-c con 0x7f5d780a24a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78141d40 con 0x7f5d780a2030 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78141bc0 con 0x7f5d7809da20 2024-09-11T05:26:33.179 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f5d78141ec0 con 0x7f5d780a24a0 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d7dbcf640 10 monclient(hunting): authenticate will time out at 2076.609574s 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a2030 0x7f5d78070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5d780a24a0 0x7f5d78070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a2030 0x7f5d78070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40772/0 (socket says 172.21.15.138:40772) 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 -- 172.21.15.138:0/3639410358 learned_addr learned my addr 172.21.15.138:0/3639410358 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient(hunting): get_auth_request con 0x7f5d780a2030 auth_method 0 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 10 monclient(hunting): get_auth_request con 0x7f5d780a24a0 auth_method 0 2024-09-11T05:26:33.180 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d777fe640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d7809da20 0x7f5d7813cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient(hunting): handle_auth_done global_id 114300 payload 824 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5d780a24a0 msgr2=0x7f5d78070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5d780a24a0 0x7f5d78070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d7809da20 msgr2=0x7f5d7813cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d7809da20 0x7f5d7813cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:26:33.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 monclient: found mon.noname-a 2024-09-11T05:26:33.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:33.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:33.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d76ffd640 1 -- 172.21.15.138:0/3639410358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d78130280 con 0x7f5d780a2030 2024-09-11T05:26:33.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.171+0000 7f5d77fff640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f5d780a24a0 0x7f5d78070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:33.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d76ffd640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a2030 msgr2=0x7f5d78070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d76ffd640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a2030 0x7f5d78070a50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00c450 tx=0x7f5d6c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 client.?.objecter ms_handle_connect 0x7f5d780a2030 2024-09-11T05:26:33.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:33.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c012a40 con 0x7f5d780a2030 2024-09-11T05:26:33.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.187 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: dump: 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: _finish_auth 0 2024-09-11T05:26:33.188 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: _check_auth_tickets 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d6c017070 con 0x7f5d780a2030 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c012e40 con 0x7f5d780a2030 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 5 monclient: authenticate success, global_id 114300 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 10 monclient: _renew_subs 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d78072510 con 0x7f5d780a2030 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 monclient: dump: 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.189 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d75ffb640 4 set_mon_vals no callback set 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 10 client.114300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 10 monclient: _renew_subs 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d780742b0 con 0x7f5d780a2030 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 1 librados: init done 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7813d100 0x7f5d780714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.190 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5d780a19a0 con 0x7f5d7813d100 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7813d100 0x7f5d780714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 10 monclient: get_auth_request con 0x7f5d7813d100 auth_method 0 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 10 monclient(hunting): handle_auth_done global_id 114303 payload 824 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 1 -- 172.21.15.138:0/3639410358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d780008e0 con 0x7f5d7813d100 2024-09-11T05:26:33.191 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f5d6c01c460 con 0x7f5d780a2030 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 1 --2- 172.21.15.138:0/3639410358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5d6403cea0 0x7f5d6403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f5d6c04f4e0 con 0x7f5d780a2030 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 client.114300.objecter ms_dispatch 0x7f5d78000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 3 client.114300.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 3 client.114300.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 20 client.114300.objecter dump_active .. 0 homeless 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d77fff640 1 --2- 172.21.15.138:0/3639410358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5d6403cea0 0x7f5d6403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d77fff640 10 monclient: get_auth_request con 0x7f5d6403cea0 auth_method 0 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7813d100 msgr2=0x7f5d780714d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 client.114300.objecter ms_handle_connect 0x7f5d7813d100 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d5ffff640 10 client.114300.objecter resend_mon_ops 2024-09-11T05:26:33.192 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.175+0000 7f5d777fe640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7813d100 0x7f5d780714d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800d3f0 tx=0x7f5d6800d8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68018070 con 0x7f5d7813d100 2024-09-11T05:26:33.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:33.193 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 10 monclient: dump: 2024-09-11T05:26:33.193 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.193 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d680070d0 con 0x7f5d7813d100 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.194 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7813d100 msgr2=0x7f5d780714d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7813d100 0x7f5d780714d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800d3f0 tx=0x7f5d6800d8b0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7813d100 msgr2=0x7f5d780714d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d7813d100 0x7f5d780714d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d5ffff640 10 client.114300.objecter ms_handle_connect 0x7f5d6403cea0 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.179+0000 7f5d77fff640 1 --2- 172.21.15.138:0/3639410358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5d6403cea0 0x7f5d6403f360 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7f5d7806d670 tx=0x7f5d700a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d7dbcf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d7dbcf640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d78133dc0 0x7f5d781341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d7dbcf640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f5d780731f0 con 0x7f5d78133dc0 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d78133dc0 0x7f5d781341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 10 monclient: get_auth_request con 0x7f5d78133dc0 auth_method 0 2024-09-11T05:26:33.195 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 10 monclient(hunting): handle_auth_done global_id 114306 payload 824 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.183+0000 7f5d777fe640 1 -- 172.21.15.138:0/3639410358 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d7813da30 con 0x7f5d78133dc0 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d777fe640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d78133dc0 msgr2=0x7f5d781341d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d777fe640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d78133dc0 0x7f5d781341d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800d3f0 tx=0x7f5d68004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 10 client.114300.objecter ms_handle_connect 0x7f5d78133dc0 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 10 client.114300.objecter resend_mon_ops 2024-09-11T05:26:33.196 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68018030 con 0x7f5d78133dc0 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 10 monclient: dump: 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.197 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 <== mon.0 v2:172.21.15.138:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f5d68023410 con 0x7f5d78133dc0 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d78133dc0 msgr2=0x7f5d781341d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d78133dc0 0x7f5d781341d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800d3f0 tx=0x7f5d68004950 comp rx=0 tx=0).stop 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d78133dc0 msgr2=0x7f5d781341d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d5ffff640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d78133dc0 0x7f5d781341d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:33.198 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "leader", 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 70, 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 265227, 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:26:33.199 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:26:33.200 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.201 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:26:33.202 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.203 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:26:33.204 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.207 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.207 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:26:33.207 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.207 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.209 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:26:33.210 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 3 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.211 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: "mgr": [ 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:26:33.212 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 10 librados: watch_flush enter 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 10 librados: watch_flush exit 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 20 client.114300.objecter shutdown clearing up homeless session... 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 10 client.114300.objecter successfully canceled tick 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5d6403cea0 msgr2=0x7f5d6403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 --2- 172.21.15.138:0/3639410358 >> v2:172.21.15.190:6800/1562782189 conn(0x7f5d6403cea0 0x7f5d6403f360 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7f5d7806d670 tx=0x7f5d700a7070 comp rx=0 tx=0).stop 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 10 monclient: shutdown 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.214 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a2030 msgr2=0x7f5d78070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 --2- 172.21.15.138:0/3639410358 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f5d780a2030 0x7f5d78070a50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00c450 tx=0x7f5d6c00c910 comp rx=0 tx=0).stop 2024-09-11T05:26:33.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d77fff640 1 -- 172.21.15.138:0/3639410358 reap_dead start 2024-09-11T05:26:33.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d77fff640 1 -- 172.21.15.138:0/3639410358 reap_dead start 2024-09-11T05:26:33.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 shutdown_connections 2024-09-11T05:26:33.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 >> 172.21.15.138:0/3639410358 conn(0x7f5d78063640 msgr2=0x7f5d78133340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 shutdown_connections 2024-09-11T05:26:33.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 -- 172.21.15.138:0/3639410358 wait complete. 2024-09-11T05:26:33.215 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.187+0000 7f5d7dbcf640 1 librados: shutdown 2024-09-11T05:26:33.215 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:26:33.215 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-11T05:26:33.216 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T05:26:33.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.331+0000 7fcb16b5d640 10 monclient: get_monmap_and_config 2024-09-11T05:26:33.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.331+0000 7fcb16b5d640 10 monclient: build_initial_monmap 2024-09-11T05:26:33.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.331+0000 7fcb16b5d640 10 monclient: monmap: 2024-09-11T05:26:33.329 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:33.329 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.329 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:33.336026+0000 2024-09-11T05:26:33.329 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:33.336026+0000 2024-09-11T05:26:33.330 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:33.330 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.330 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:33.330 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:33.330 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:33.330 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:33.330 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:33.330 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.331+0000 7fcb16b5d640 1 Processor -- start 2024-09-11T05:26:33.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.331+0000 7fcb16b5d640 1 -- start start 2024-09-11T05:26:33.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.331+0000 7fcb16b5d640 10 monclient: init 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb1009da20 0x7fcb100a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb1009da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb100a1e60 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fcb100a22a0 0x7fcb10136910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb100a22a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb10064590 con 0x7fcb100a1e60 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fcb1009de00 con 0x7fcb100a22a0 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcb100a2c70 con 0x7fcb1009da20 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:33.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb16b5d640 10 monclient(hunting): authenticate will time out at 2076.772258s 2024-09-11T05:26:33.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb1009da20 0x7fcb100a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fcb100a22a0 0x7fcb10136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb1009da20 0x7fcb100a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55528/0 (socket says 172.21.15.138:55528) 2024-09-11T05:26:33.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient(hunting): get_auth_request con 0x7fcb100a22a0 auth_method 0 2024-09-11T05:26:33.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 1 -- 172.21.15.138:0/3280748703 learned_addr learned my addr 172.21.15.138:0/3280748703 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:33.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb0ffff640 1 --1- 172.21.15.138:0/3280748703 >> v1:172.21.15.138:6789/0 conn(0x7fcb100a1e60 0x7fcb10136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:49990/0 (socket says 172.21.15.138:49990) 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 10 monclient(hunting): get_auth_request con 0x7fcb1009da20 auth_method 0 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient(hunting): handle_auth_done global_id 114309 payload 274 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 1 -- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb1009da20 msgr2=0x7fcb100a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 1 --2- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb1009da20 0x7fcb100a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 1 -- 172.21.15.138:0/3280748703 >> v1:172.21.15.138:6789/0 conn(0x7fcb100a1e60 legacy=0x7fcb10136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient: _finish_hunting 0 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 1 monclient: found mon.noname-a 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:33.336 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb150fa640 1 -- 172.21.15.138:0/3280748703 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb100a2f50 con 0x7fcb100a22a0 2024-09-11T05:26:33.337 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.335+0000 7fcb148f9640 1 --2- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb1009da20 0x7fcb100a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:33.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 -- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb100a22a0 msgr2=0x7fcb10136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 --2- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb100a22a0 0x7fcb10136910 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb000097c0 tx=0x7fcb0001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cdfa3c5e995f29 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 1 -- 172.21.15.138:0/3280748703 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000b0070 con 0x7fcb100a22a0 2024-09-11T05:26:33.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:33.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: dump: 2024-09-11T05:26:33.345 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.345 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.345 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.345 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.345 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 1 -- 172.21.15.138:0/3280748703 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb000ab600 con 0x7fcb100a22a0 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 1 -- 172.21.15.138:0/3280748703 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000ab9d0 con 0x7fcb100a22a0 2024-09-11T05:26:33.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb0f7fe640 10 monclient: dump: 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 5 monclient: authenticate success, global_id 114309 2024-09-11T05:26:33.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 4 set_mon_vals no callback set 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient: shutdown 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb100a22a0 msgr2=0x7fcb10136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 --2- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb100a22a0 0x7fcb10136910 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb000097c0 tx=0x7fcb0001c360 comp rx=0 tx=0).stop 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- 172.21.15.138:0/3280748703 shutdown_connections 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 --2- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb100a22a0 0x7fcb10136910 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 --2- 172.21.15.138:0/3280748703 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb1009da20 0x7fcb100a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- 172.21.15.138:0/3280748703 >> 172.21.15.138:0/3280748703 conn(0x7fcb10063640 msgr2=0x7fcb10063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- 172.21.15.138:0/3280748703 shutdown_connections 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- 172.21.15.138:0/3280748703 wait complete. 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient: build_initial_monmap 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient: monmap: 2024-09-11T05:26:33.348 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 librados: starting msgr at 2024-09-11T05:26:33.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 librados: starting objecter 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 Processor -- start 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- start start 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 librados: setting wanted keys 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 librados: calling monclient init 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient: init 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb1009da20 0x7fcb100709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb1009da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb100a1e60 0x7fcb10070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb100a1e60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb100a22a0 0x7fcb101ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb100a22a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcb10140ee0 con 0x7fcb1009da20 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb10140d60 con 0x7fcb100a1e60 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcb10141060 con 0x7fcb100a22a0 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb16b5d640 10 monclient(hunting): authenticate will time out at 2076.779521s 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb1009da20 0x7fcb100709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb100a22a0 0x7fcb101ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb1009da20 0x7fcb100709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40808/0 (socket says 172.21.15.138:40808) 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 1 -- 172.21.15.138:0/1511803816 learned_addr learned my addr 172.21.15.138:0/1511803816 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 10 monclient(hunting): get_auth_request con 0x7fcb1009da20 auth_method 0 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient(hunting): get_auth_request con 0x7fcb100a22a0 auth_method 0 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient(hunting): handle_auth_done global_id 84602 payload 824 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb100a1e60 msgr2=0x7fcb10070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb100a1e60 0x7fcb10070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb1009da20 msgr2=0x7fcb100709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb1009da20 0x7fcb100709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient: _finish_hunting 0 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 monclient: found mon.noname-c 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb150fa640 1 -- 172.21.15.138:0/1511803816 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1009e110 con 0x7fcb100a22a0 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.339+0000 7fcb148f9640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb1009da20 0x7fcb100709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb150fa640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb100a22a0 msgr2=0x7fcb101ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb150fa640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb100a22a0 0x7fcb101ce410 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcb000ad000 tx=0x7fcb00019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 client.?.objecter ms_handle_connect 0x7fcb100a22a0 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000b9020 con 0x7fcb100a22a0 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: dump: 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: _finish_auth 0 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:26:33.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb000b5690 con 0x7fcb100a22a0 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000b0040 con 0x7fcb100a22a0 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 monclient: dump: 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.355 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 5 monclient: authenticate success, global_id 84602 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 10 monclient: _renew_subs 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0effd640 4 set_mon_vals no callback set 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb10072270 con 0x7fcb100a22a0 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 10 client.84602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 10 monclient: _renew_subs 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb10134900 con 0x7fcb100a22a0 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 1 librados: init done 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb10073ad0 0x7fcb101ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcb10073eb0 con 0x7fcb10073ad0 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb10073ad0 0x7fcb101ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 10 monclient: get_auth_request con 0x7fcb10073ad0 auth_method 0 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fcb000b5aa0 con 0x7fcb100a22a0 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 10 monclient(hunting): handle_auth_done global_id 84605 payload 824 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb148f9640 1 -- 172.21.15.138:0/1511803816 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb10136b90 con 0x7fcb10073ad0 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 1 --2- 172.21.15.138:0/1511803816 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcafc03cef0 0x7fcafc03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7fcb000e9950 con 0x7fcb100a22a0 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 10 client.84602.objecter ms_dispatch 0x7fcb10000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 3 client.84602.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 3 client.84602.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:33.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.343+0000 7fcb0d7fa640 20 client.84602.objecter dump_active .. 0 homeless 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0ffff640 1 --2- 172.21.15.138:0/1511803816 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcafc03cef0 0x7fcafc03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0ffff640 10 monclient: get_auth_request con 0x7fcafc03cef0 auth_method 0 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 10 client.84602.objecter ms_handle_connect 0x7fcb10073ad0 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 10 client.84602.objecter resend_mon_ops 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb148f9640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb10073ad0 msgr2=0x7fcb101ce950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb148f9640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb10073ad0 0x7fcb101ce950 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb08004910 tx=0x7fcb0800e840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08007800 con 0x7fcb10073ad0 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 10 monclient: dump: 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.359 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.360 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.360 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.360 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.360 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.360 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb0801b070 con 0x7fcb10073ad0 2024-09-11T05:26:33.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:33.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb10073ad0 msgr2=0x7fcb101ce950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb10073ad0 0x7fcb101ce950 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb08004910 tx=0x7fcb0800e840 comp rx=0 tx=0).stop 2024-09-11T05:26:33.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb10073ad0 msgr2=0x7fcb101ce950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:26:33.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb10073ad0 0x7fcb101ce950 secure :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb08004910 tx=0x7fcb0800e840 comp rx=0 tx=0).stop 2024-09-11T05:26:33.362 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0d7fa640 10 client.84602.objecter ms_handle_connect 0x7fcafc03cef0 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.347+0000 7fcb0ffff640 1 --2- 172.21.15.138:0/1511803816 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcafc03cef0 0x7fcafc03f3b0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7fcb1009e050 tx=0x7fcb040079f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb101cee90 0x7fcb1012daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcb100a1920 con 0x7fcb101cee90 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb101cee90 0x7fcb1012daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 10 monclient: get_auth_request con 0x7fcb101cee90 auth_method 0 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 10 monclient(hunting): handle_auth_done global_id 84608 payload 824 2024-09-11T05:26:33.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 1 -- 172.21.15.138:0/1511803816 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb1006cca0 con 0x7fcb101cee90 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb101cee90 msgr2=0x7fcb1012daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb148f9640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb101cee90 0x7fcb1012daa0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb0801a000 tx=0x7fcb08004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 10 client.84602.objecter ms_handle_connect 0x7fcb101cee90 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 10 client.84602.objecter resend_mon_ops 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08016670 con 0x7fcb101cee90 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 10 monclient: dump: 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.364 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 <== mon.2 v2:172.21.15.138:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fcb08016670 con 0x7fcb101cee90 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb101cee90 msgr2=0x7fcb1012daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb101cee90 0x7fcb1012daa0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb0801a000 tx=0x7fcb08004a50 comp rx=0 tx=0).stop 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb101cee90 msgr2=0x7fcb1012daa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:26:33.365 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb0d7fa640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb101cee90 0x7fcb1012daa0 secure :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb0801a000 tx=0x7fcb08004a50 comp rx=0 tx=0).stop 2024-09-11T05:26:33.366 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:33.366 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:26:33.366 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:26:33.366 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:26:33.366 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 70, 2024-09-11T05:26:33.366 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 637125, 2024-09-11T05:26:33.366 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:26:33.367 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:26:33.368 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:26:33.370 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.371 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:26:33.372 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:26:33.374 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:26:33.376 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:26:33.377 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: "osd": [ 2024-09-11T05:26:33.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 6 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 5 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.381 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:26:33.381 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 10 librados: watch_flush enter 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 10 librados: watch_flush exit 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 20 client.84602.objecter shutdown clearing up homeless session... 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 10 client.84602.objecter successfully canceled tick 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcafc03cef0 msgr2=0x7fcafc03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 1 --2- 172.21.15.138:0/1511803816 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcafc03cef0 0x7fcafc03f3b0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7fcb1009e050 tx=0x7fcb040079f0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 10 monclient: shutdown 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb100a22a0 msgr2=0x7fcb101ce410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb16b5d640 1 --2- 172.21.15.138:0/1511803816 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb100a22a0 0x7fcb101ce410 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcb000ad000 tx=0x7fcb00019100 comp rx=0 tx=0).stop 2024-09-11T05:26:33.383 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb150fa640 1 -- 172.21.15.138:0/1511803816 reap_dead start 2024-09-11T05:26:33.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.355+0000 7fcb150fa640 1 -- 172.21.15.138:0/1511803816 reap_dead start 2024-09-11T05:26:33.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.359+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 shutdown_connections 2024-09-11T05:26:33.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.359+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 >> 172.21.15.138:0/1511803816 conn(0x7fcb10063640 msgr2=0x7fcb100a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.359+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 shutdown_connections 2024-09-11T05:26:33.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.359+0000 7fcb16b5d640 1 -- 172.21.15.138:0/1511803816 wait complete. 2024-09-11T05:26:33.384 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.359+0000 7fcb16b5d640 1 librados: shutdown 2024-09-11T05:26:33.384 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:26:33.384 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-11T05:26:33.384 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T05:26:33.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient: get_monmap_and_config 2024-09-11T05:26:33.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient: build_initial_monmap 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient: monmap: 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:33.501953+0000 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:33.501953+0000 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:33.496 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 1 Processor -- start 2024-09-11T05:26:33.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 1 -- start start 2024-09-11T05:26:33.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient: init 2024-09-11T05:26:33.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:33.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-11T05:26:33.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fba8409da20 0x7fba840a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient(hunting): picked mon.noname-c con 0x7fba8409da20 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba840a1e60 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a22d0 0x7fba84136910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient(hunting): picked mon.noname-b con 0x7fba840a22d0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba84064590 con 0x7fba840a1e60 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fba8409de00 con 0x7fba8409da20 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fba840a2700 con 0x7fba840a22d0 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba8ac57640 10 monclient(hunting): authenticate will time out at 2076.937733s 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a22d0 0x7fba84136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a22d0 0x7fba84136910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55556/0 (socket says 172.21.15.138:55556) 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 1 -- 172.21.15.138:0/1628933777 learned_addr learned my addr 172.21.15.138:0/1628933777 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient(hunting): get_auth_request con 0x7fba840a22d0 auth_method 0 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba889f3640 1 --2- 172.21.15.138:0/1628933777 >> v2:172.21.15.190:3300/0 conn(0x7fba8409da20 0x7fba840a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient(hunting): handle_auth_done global_id 84611 payload 274 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 1 -- 172.21.15.138:0/1628933777 >> v2:172.21.15.190:3300/0 conn(0x7fba8409da20 msgr2=0x7fba840a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:33.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 1 --2- 172.21.15.138:0/1628933777 >> v2:172.21.15.190:3300/0 conn(0x7fba8409da20 0x7fba840a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 1 -- 172.21.15.138:0/1628933777 >> v1:172.21.15.190:6789/0 conn(0x7fba840a1e60 legacy=0x7fba84136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:33.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient: _finish_hunting 0 2024-09-11T05:26:33.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 1 monclient: found mon.noname-b 2024-09-11T05:26:33.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:33.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba891f4640 1 -- 172.21.15.138:0/1628933777 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba840a2c70 con 0x7fba840a22d0 2024-09-11T05:26:33.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba889f3640 10 monclient: get_auth_request con 0x7fba8409da20 auth_method 0 2024-09-11T05:26:33.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.499+0000 7fba889f3640 1 --2- 172.21.15.138:0/1628933777 >> v2:172.21.15.190:3300/0 conn(0x7fba8409da20 0x7fba840a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:26:33.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba891f4640 1 -- 172.21.15.138:0/1628933777 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a22d0 msgr2=0x7fba84136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba891f4640 1 --2- 172.21.15.138:0/1628933777 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a22d0 0x7fba84136910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009e50 tx=0x7fba7c01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eba3a058585d66ed server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 1 -- 172.21.15.138:0/1628933777 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0a56f0 con 0x7fba840a22d0 2024-09-11T05:26:33.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:26:33.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: dump: 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: _finish_auth 0 2024-09-11T05:26:33.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: _check_auth_tickets 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 1 -- 172.21.15.138:0/1628933777 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba7c0a5890 con 0x7fba840a22d0 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 1 -- 172.21.15.138:0/1628933777 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0a5c60 con 0x7fba840a22d0 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 5 monclient: authenticate success, global_id 84611 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 4 set_mon_vals no callback set 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba837fe640 10 monclient: dump: 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.514 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 10 monclient: shutdown 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 1 -- 172.21.15.138:0/1628933777 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a22d0 msgr2=0x7fba84136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 1 --2- 172.21.15.138:0/1628933777 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a22d0 0x7fba84136910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009e50 tx=0x7fba7c01c8f0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.515 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 1 -- 172.21.15.138:0/1628933777 shutdown_connections 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 1 --2- 172.21.15.138:0/1628933777 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a22d0 0x7fba84136910 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 1 --2- 172.21.15.138:0/1628933777 >> v2:172.21.15.190:3300/0 conn(0x7fba8409da20 0x7fba840a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 1 -- 172.21.15.138:0/1628933777 >> 172.21.15.138:0/1628933777 conn(0x7fba84063640 msgr2=0x7fba84063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 1 -- 172.21.15.138:0/1628933777 shutdown_connections 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 1 -- 172.21.15.138:0/1628933777 wait complete. 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 10 monclient: build_initial_monmap 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.503+0000 7fba8ac57640 10 monclient: monmap: 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:33.516 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 librados: starting msgr at 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 librados: starting objecter 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 Processor -- start 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 -- start start 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 librados: setting wanted keys 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 librados: calling monclient init 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient: init 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba8409da20 0x7fba8413dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient(hunting): picked mon.noname-a con 0x7fba8409da20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a1e60 0x7fba8409e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient(hunting): picked mon.noname-c con 0x7fba840a1e60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba840a22d0 0x7fba8409e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient(hunting): picked mon.noname-b con 0x7fba840a22d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fba84140ee0 con 0x7fba8409da20 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fba84140d60 con 0x7fba840a22d0 2024-09-11T05:26:33.518 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fba84141060 con 0x7fba840a1e60 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba8ac57640 10 monclient(hunting): authenticate will time out at 2076.944863s 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba8409da20 0x7fba8413dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a1e60 0x7fba8409e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba8409da20 0x7fba8413dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40818/0 (socket says 172.21.15.138:40818) 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 -- 172.21.15.138:0/295582752 learned_addr learned my addr 172.21.15.138:0/295582752 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient(hunting): get_auth_request con 0x7fba8409da20 auth_method 0 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 10 monclient(hunting): get_auth_request con 0x7fba840a1e60 auth_method 0 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.519 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba891f4640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba840a22d0 0x7fba8409e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba891f4640 10 monclient(hunting): get_auth_request con 0x7fba840a22d0 auth_method 0 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba891f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba891f4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba891f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient(hunting): handle_auth_done global_id 114315 payload 824 2024-09-11T05:26:33.520 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a1e60 msgr2=0x7fba8409e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a1e60 0x7fba8409e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba840a22d0 msgr2=0x7fba8409e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba840a22d0 0x7fba8409e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient: _finish_hunting 0 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 monclient: found mon.noname-a 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 -- 172.21.15.138:0/295582752 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8413d2d0 con 0x7fba8409da20 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba83fff640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fba840a1e60 0x7fba8409e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba8409da20 msgr2=0x7fba8413dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.521 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba889f3640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba8409da20 0x7fba8413dd40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c7c0 tx=0x7fba7400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 client.?.objecter ms_handle_connect 0x7fba8409da20 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74007be0 con 0x7fba8409da20 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: dump: 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.522 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: _finish_auth 0 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: _check_auth_tickets 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba74007d80 con 0x7fba8409da20 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74011630 con 0x7fba8409da20 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba817fa640 10 monclient: dump: 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.523 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.507+0000 7fba82ffd640 4 set_mon_vals no callback set 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 5 monclient: authenticate success, global_id 114315 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 10 monclient: _renew_subs 2024-09-11T05:26:33.524 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8409e110 con 0x7fba8409da20 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 10 client.114315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 10 monclient: _renew_subs 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8413cff0 con 0x7fba8409da20 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 1 librados: init done 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba84073ec0 0x7fba8409eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fba84134360 con 0x7fba84073ec0 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fba740117d0 con 0x7fba8409da20 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba817fa640 1 --2- 172.21.15.138:0/295582752 >> v2:172.21.15.190:6800/1562782189 conn(0x7fba7003cea0 0x7fba7003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba84073ec0 0x7fba8409eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.525 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7fba7404fa90 con 0x7fba8409da20 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba817fa640 10 client.114315.objecter ms_dispatch 0x7fba84000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba817fa640 3 client.114315.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba817fa640 3 client.114315.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 10 monclient: get_auth_request con 0x7fba84073ec0 auth_method 0 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba817fa640 20 client.114315.objecter dump_active .. 0 homeless 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba891f4640 1 --2- 172.21.15.138:0/295582752 >> v2:172.21.15.190:6800/1562782189 conn(0x7fba7003cea0 0x7fba7003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba891f4640 10 monclient: get_auth_request con 0x7fba7003cea0 auth_method 0 2024-09-11T05:26:33.526 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 10 monclient(hunting): handle_auth_done global_id 134104 payload 824 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.511+0000 7fba83fff640 1 -- 172.21.15.138:0/295582752 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba840a1920 con 0x7fba84073ec0 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 10 client.114315.objecter ms_handle_connect 0x7fba84073ec0 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 10 client.114315.objecter resend_mon_ops 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba83fff640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba84073ec0 msgr2=0x7fba8409eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba83fff640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba84073ec0 0x7fba8409eed0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba840a0600 tx=0x7fba7800ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 10 client.114315.objecter ms_handle_connect 0x7fba7003cea0 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba891f4640 1 --2- 172.21.15.138:0/295582752 >> v2:172.21.15.190:6800/1562782189 conn(0x7fba7003cea0 0x7fba7003f360 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7fba8409e050 tx=0x7fba7c0a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba78018020 con 0x7fba84073ec0 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 10 monclient: dump: 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.527 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fba7800dd30 con 0x7fba84073ec0 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba84073ec0 msgr2=0x7fba8409eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.528 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba84073ec0 0x7fba8409eed0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba840a0600 tx=0x7fba7800ff70 comp rx=0 tx=0).stop 2024-09-11T05:26:33.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba84073ec0 msgr2=0x7fba8409eed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:26:33.529 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.515+0000 7fba817fa640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba84073ec0 0x7fba8409eed0 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba840a0600 tx=0x7fba7800ff70 comp rx=0 tx=0).stop 2024-09-11T05:26:33.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba8ac57640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T05:26:33.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba8ac57640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba8412d910 0x7fba8412dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.530 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba8ac57640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fba8413bad0 con 0x7fba8412d910 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba83fff640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba8412d910 0x7fba8412dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba83fff640 10 monclient: get_auth_request con 0x7fba8412d910 auth_method 0 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba83fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.519+0000 7fba83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.531 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba83fff640 10 monclient(hunting): handle_auth_done global_id 134107 payload 824 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba83fff640 1 -- 172.21.15.138:0/295582752 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fba8406cca0 con 0x7fba8412d910 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba83fff640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba8412d910 msgr2=0x7fba8412dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba83fff640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba8412d910 0x7fba8412dd20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fba78019000 tx=0x7fba78003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 10 client.114315.objecter ms_handle_connect 0x7fba8412d910 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 10 client.114315.objecter resend_mon_ops 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba78018030 con 0x7fba8412d910 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 10 monclient: dump: 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.532 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fba78014b80 con 0x7fba8412d910 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba8412d910 msgr2=0x7fba8412dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.533 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba8412d910 0x7fba8412dd20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fba78019000 tx=0x7fba78003700 comp rx=0 tx=0).stop 2024-09-11T05:26:33.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba8412d910 msgr2=0x7fba8412dd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T05:26:33.534 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba817fa640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba8412d910 0x7fba8412dd20 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fba78019000 tx=0x7fba78003700 comp rx=0 tx=0).stop 2024-09-11T05:26:33.534 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:33.534 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:26:33.534 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:26:33.534 INFO:teuthology.orchestra.run.smithi138.stdout: "state": "peon", 2024-09-11T05:26:33.534 INFO:teuthology.orchestra.run.smithi138.stdout: "election_epoch": 70, 2024-09-11T05:26:33.534 INFO:teuthology.orchestra.run.smithi138.stdout: "uptime": 15779, 2024-09-11T05:26:33.535 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum": [ 2024-09-11T05:26:33.535 INFO:teuthology.orchestra.run.smithi138.stdout: 0, 2024-09-11T05:26:33.535 INFO:teuthology.orchestra.run.smithi138.stdout: 1, 2024-09-11T05:26:33.535 INFO:teuthology.orchestra.run.smithi138.stdout: 2 2024-09-11T05:26:33.535 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.535 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_age": 0, 2024-09-11T05:26:33.535 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "required_con": "2450521708007866372", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "required_mon": [ 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.536 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_con": "4540719139924082687", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "quorum_mon": [ 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.537 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "outside_quorum": [], 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "extra_probe_peers": [], 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "sync_provider": [], 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "monmap": { 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "epoch": 1, 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "fsid": "66c6f018-f82d-4af3-8f0d-6530cdae2ec3", 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "modified": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:26:33.538 INFO:teuthology.orchestra.run.smithi138.stdout: "created": "2024-09-11T05:06:55.201195Z", 2024-09-11T05:26:33.539 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release": 19, 2024-09-11T05:26:33.539 INFO:teuthology.orchestra.run.smithi138.stdout: "min_mon_release_name": "squid", 2024-09-11T05:26:33.539 INFO:teuthology.orchestra.run.smithi138.stdout: "election_strategy": 1, 2024-09-11T05:26:33.539 INFO:teuthology.orchestra.run.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-11T05:26:33.539 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false, 2024-09-11T05:26:33.539 INFO:teuthology.orchestra.run.smithi138.stdout: "tiebreaker_mon": "", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "removed_ranks: ": "", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "features": { 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "persistent": [ 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "kraken", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "luminous", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "mimic", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "osdmap-prune", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "nautilus", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "octopus", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "pacific", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "elector-pinging", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "quincy", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "reef", 2024-09-11T05:26:33.540 INFO:teuthology.orchestra.run.smithi138.stdout: "squid" 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: "optional": [] 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: "mons": [ 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 0, 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "a", 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.541 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3300", 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789", 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6789/0", 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6789/0", 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.542 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 1, 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "b", 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:3300", 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.543 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789", 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "rank": 2, 2024-09-11T05:26:33.544 INFO:teuthology.orchestra.run.smithi138.stdout: "name": "c", 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addrs": { 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: "addrvec": [ 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v2", 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:3301", 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: "type": "v1", 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790", 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: "nonce": 0 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.545 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.546 INFO:teuthology.orchestra.run.smithi138.stdout: "addr": "172.21.15.138:6790/0", 2024-09-11T05:26:33.546 INFO:teuthology.orchestra.run.smithi138.stdout: "public_addr": "172.21.15.138:6790/0", 2024-09-11T05:26:33.546 INFO:teuthology.orchestra.run.smithi138.stdout: "priority": 0, 2024-09-11T05:26:33.546 INFO:teuthology.orchestra.run.smithi138.stdout: "weight": 0, 2024-09-11T05:26:33.546 INFO:teuthology.orchestra.run.smithi138.stdout: "crush_location": "{}" 2024-09-11T05:26:33.546 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.546 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "feature_map": { 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "mon": [ 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "client": [ 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "release": "squid", 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: "num": 1 2024-09-11T05:26:33.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-09-11T05:26:33.548 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-09-11T05:26:33.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-09-11T05:26:33.548 INFO:teuthology.orchestra.run.smithi138.stdout: "stretch_mode": false 2024-09-11T05:26:33.548 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba8ac57640 10 librados: watch_flush enter 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba8ac57640 10 librados: watch_flush exit 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba8ac57640 20 client.114315.objecter shutdown clearing up homeless session... 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.523+0000 7fba8ac57640 10 client.114315.objecter successfully canceled tick 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 >> v2:172.21.15.190:6800/1562782189 conn(0x7fba7003cea0 msgr2=0x7fba7003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 --2- 172.21.15.138:0/295582752 >> v2:172.21.15.190:6800/1562782189 conn(0x7fba7003cea0 0x7fba7003f360 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7fba8409e050 tx=0x7fba7c0a7070 comp rx=0 tx=0).stop 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 10 monclient: shutdown 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba8409da20 msgr2=0x7fba8413dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba8409da20 0x7fba8413dd40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c7c0 tx=0x7fba7400cc80 comp rx=0 tx=0).stop 2024-09-11T05:26:33.550 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba891f4640 1 -- 172.21.15.138:0/295582752 reap_dead start 2024-09-11T05:26:33.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 shutdown_connections 2024-09-11T05:26:33.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 --2- 172.21.15.138:0/295582752 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fba8409da20 0x7fba8413dd40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 >> 172.21.15.138:0/295582752 conn(0x7fba84063640 msgr2=0x7fba84136380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 shutdown_connections 2024-09-11T05:26:33.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 -- 172.21.15.138:0/295582752 wait complete. 2024-09-11T05:26:33.551 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.527+0000 7fba8ac57640 1 librados: shutdown 2024-09-11T05:26:33.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-11T05:26:33.551 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T05:26:33.551 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T05:26:33.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient: get_monmap_and_config 2024-09-11T05:26:33.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient: build_initial_monmap 2024-09-11T05:26:33.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient: monmap: 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:33.663413+0000 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:33.663413+0000 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:33.657 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 1 Processor -- start 2024-09-11T05:26:33.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 1 -- start start 2024-09-11T05:26:33.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient: init 2024-09-11T05:26:33.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19d40a5b40 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d40a7130 0x7f19d413db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient(hunting): picked mon.noname-b con 0x7f19d40a7130 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19d409d840 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d409d490 con 0x7f19d409d840 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d4070580 con 0x7f19d40a5b40 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f19d409e110 con 0x7f19d40a7130 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:33.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19da23d640 10 monclient(hunting): authenticate will time out at 2077.099410s 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d40a7130 0x7f19d413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d40a7130 0x7f19d413db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55572/0 (socket says 172.21.15.138:55572) 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 1 -- 172.21.15.138:0/3435392733 learned_addr learned my addr 172.21.15.138:0/3435392733 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient(hunting): get_auth_request con 0x7f19d40a7130 auth_method 0 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d3fff640 1 --1- 172.21.15.138:0/3435392733 >> v1:172.21.15.138:6789/0 conn(0x7f19d409d840 0x7f19d4140090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50000/0 (socket says 172.21.15.138:50000) 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient(hunting): handle_auth_done global_id 84617 payload 274 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 1 -- 172.21.15.138:0/3435392733 >> v1:172.21.15.190:6789/0 conn(0x7f19d40a5b40 legacy=0x7f19d40a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 1 -- 172.21.15.138:0/3435392733 >> v1:172.21.15.138:6789/0 conn(0x7f19d409d840 legacy=0x7f19d4140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient: _finish_hunting 0 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 1 monclient: found mon.noname-b 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.659+0000 7f19d2ffd640 1 -- 172.21.15.138:0/3435392733 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d409cb80 con 0x7f19d40a7130 2024-09-11T05:26:33.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d2ffd640 1 -- 172.21.15.138:0/3435392733 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d40a7130 msgr2=0x7f19d413db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d2ffd640 1 --2- 172.21.15.138:0/3435392733 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d40a7130 0x7f19d413db40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f19c80098a0 tx=0x7f19c801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17478975560535ff server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 1 -- 172.21.15.138:0/3435392733 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c80b0070 con 0x7f19d40a7130 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: dump: 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.665 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: _finish_auth 0 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: _check_auth_tickets 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 1 -- 172.21.15.138:0/3435392733 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c80ab600 con 0x7f19d40a7130 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 5 monclient: authenticate success, global_id 84617 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 1 -- 172.21.15.138:0/3435392733 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c80ab9d0 con 0x7f19d40a7130 2024-09-11T05:26:33.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 4 set_mon_vals no callback set 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d27fc640 10 monclient: dump: 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient: shutdown 2024-09-11T05:26:33.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- 172.21.15.138:0/3435392733 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d40a7130 msgr2=0x7f19d413db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 --2- 172.21.15.138:0/3435392733 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d40a7130 0x7f19d413db40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f19c80098a0 tx=0x7f19c801c440 comp rx=0 tx=0).stop 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- 172.21.15.138:0/3435392733 shutdown_connections 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 --2- 172.21.15.138:0/3435392733 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d40a7130 0x7f19d413db40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- 172.21.15.138:0/3435392733 >> 172.21.15.138:0/3435392733 conn(0x7f19d40aba10 msgr2=0x7f19d40a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- 172.21.15.138:0/3435392733 shutdown_connections 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- 172.21.15.138:0/3435392733 wait complete. 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient: build_initial_monmap 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient: monmap: 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:33.668 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 librados: starting msgr at 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 librados: starting objecter 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 Processor -- start 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- start start 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 librados: setting wanted keys 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 librados: calling monclient init 2024-09-11T05:26:33.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient: init 2024-09-11T05:26:33.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:33.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:26:33.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d409d840 0x7f19d4140980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient(hunting): picked mon.noname-c con 0x7f19d409d840 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:33.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19d40a5b40 0x7f19d41d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.674 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient(hunting): picked mon.noname-b con 0x7f19d40a5b40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19d40a7130 0x7f19d413e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient(hunting): picked mon.noname-a con 0x7f19d40a7130 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f19d4144360 con 0x7f19d40a7130 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f19d41441e0 con 0x7f19d40a5b40 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f19d41444e0 con 0x7f19d409d840 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19da23d640 10 monclient(hunting): authenticate will time out at 2077.106684s 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d409d840 0x7f19d4140980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d409d840 0x7f19d4140980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55582/0 (socket says 172.21.15.138:55582) 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 -- 172.21.15.138:0/1406994601 learned_addr learned my addr 172.21.15.138:0/1406994601 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:33.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient(hunting): get_auth_request con 0x7f19d409d840 auth_method 0 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d3fff640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19d40a7130 0x7f19d413e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d3fff640 10 monclient(hunting): get_auth_request con 0x7f19d40a7130 auth_method 0 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:33.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d2ffd640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19d40a5b40 0x7f19d41d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient(hunting): handle_auth_done global_id 84620 payload 824 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 -- 172.21.15.138:0/1406994601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19d40a5b40 msgr2=0x7f19d41d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19d40a5b40 0x7f19d41d0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.677 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 -- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19d40a7130 msgr2=0x7f19d413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19d40a7130 0x7f19d413e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient: _finish_hunting 0 2024-09-11T05:26:33.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 monclient: found mon.noname-c 2024-09-11T05:26:33.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:33.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d37fe640 1 -- 172.21.15.138:0/1406994601 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d4122160 con 0x7f19d409d840 2024-09-11T05:26:33.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d2ffd640 10 monclient: get_auth_request con 0x7f19d40a5b40 auth_method 0 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.667+0000 7f19d2ffd640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19d40a5b40 0x7f19d41d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19d37fe640 1 -- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d409d840 msgr2=0x7f19d4140980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19d37fe640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d409d840 0x7f19d4140980 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f19c400c450 tx=0x7f19c400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 client.?.objecter ms_handle_connect 0x7f19d409d840 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 1 -- 172.21.15.138:0/1406994601 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c4012a40 con 0x7f19d409d840 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: dump: 2024-09-11T05:26:33.680 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: _finish_auth 0 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: _check_auth_tickets 2024-09-11T05:26:33.681 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 1 -- 172.21.15.138:0/1406994601 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c4017070 con 0x7f19d409d840 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 5 monclient: authenticate success, global_id 84620 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 10 monclient: _renew_subs 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d40a2330 con 0x7f19d409d840 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 1 -- 172.21.15.138:0/1406994601 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c4012e40 con 0x7f19d409d840 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:33.682 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 monclient: dump: 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 10 client.84620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 10 monclient: _renew_subs 2024-09-11T05:26:33.683 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19d17fa640 4 set_mon_vals no callback set 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d4121e80 con 0x7f19d409d840 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 1 librados: init done 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d40a5f20 con 0x7f19d409d840 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 1 -- 172.21.15.138:0/1406994601 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f19c401c460 con 0x7f19d409d840 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 1 --2- 172.21.15.138:0/1406994601 >> v2:172.21.15.190:6800/1562782189 conn(0x7f19c003cea0 0x7f19c003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:33.684 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 1 -- 172.21.15.138:0/1406994601 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f19c404f490 con 0x7f19d409d840 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 10 client.84620.objecter ms_dispatch 0x7f19d4000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 3 client.84620.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 3 client.84620.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19bbfff640 20 client.84620.objecter dump_active .. 0 homeless 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19d2ffd640 1 --2- 172.21.15.138:0/1406994601 >> v2:172.21.15.190:6800/1562782189 conn(0x7f19c003cea0 0x7f19c003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.671+0000 7f19d2ffd640 10 monclient: get_auth_request con 0x7f19c003cea0 auth_method 0 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.675+0000 7f19d2ffd640 1 --2- 172.21.15.138:0/1406994601 >> v2:172.21.15.190:6800/1562782189 conn(0x7f19c003cea0 0x7f19c003f360 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7f19d406d5a0 tx=0x7f19c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.675+0000 7f19bbfff640 10 client.84620.objecter ms_handle_connect 0x7f19c003cea0 2024-09-11T05:26:33.685 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.679+0000 7f19bbfff640 1 -- 172.21.15.138:0/1406994601 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f19c401ac60 con 0x7f19d409d840 2024-09-11T05:26:33.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.679+0000 7f19bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:33.686 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.679+0000 7f19bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:33.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.895+0000 7f19da23d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T05:26:33.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.895+0000 7f19da23d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T05:26:33.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.895+0000 7f19da23d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:33.895 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.895+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f19d4138e80 con 0x7f19d409d840 2024-09-11T05:26:33.898 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.898 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=8,logm=58} crc {auth=2959455537,config=1228700967,health=1674371620,kv=1961940245,logm=1790624584}) 2024-09-11T05:26:33.899 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=11,kv=10,logm=54} crc {auth=2146688854,config=692619927,health=695325213,kv=310889196,logm=1033906774}) 2024-09-11T05:26:33.899 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19bbfff640 1 -- 172.21.15.138:0/1406994601 <== mon.2 v2:172.21.15.138:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f19c401c960 con 0x7f19d409d840 2024-09-11T05:26:33.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T05:26:33.899 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:26:33.900 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=8,logm=58} crc {auth=2959455537,config=1228700967,health=1674371620,kv=1961940245,logm=1790624584}) 2024-09-11T05:26:33.900 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=11,kv=10,logm=56} crc {auth=829655940,config=692619927,health=3304737543,kv=310889197,logm=4106995964}) 2024-09-11T05:26:33.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 10 librados: watch_flush enter 2024-09-11T05:26:33.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 10 librados: watch_flush exit 2024-09-11T05:26:33.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 20 client.84620.objecter shutdown clearing up homeless session... 2024-09-11T05:26:33.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 10 client.84620.objecter successfully canceled tick 2024-09-11T05:26:33.902 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 >> v2:172.21.15.190:6800/1562782189 conn(0x7f19c003cea0 msgr2=0x7f19c003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 --2- 172.21.15.138:0/1406994601 >> v2:172.21.15.190:6800/1562782189 conn(0x7f19c003cea0 0x7f19c003f360 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7f19d406d5a0 tx=0x7f19c8008040 comp rx=0 tx=0).stop 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 10 monclient: shutdown 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d409d840 msgr2=0x7f19d4140980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d409d840 0x7f19d4140980 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f19c400c450 tx=0x7f19c400c910 comp rx=0 tx=0).stop 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 shutdown_connections 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 --2- 172.21.15.138:0/1406994601 >> v2:172.21.15.190:6800/1562782189 conn(0x7f19c003cea0 0x7f19c003f360 unknown :-1 s=CLOSED pgs=3517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f19d40a7130 0x7f19d413e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19d40a5b40 0x7f19d41d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 --2- 172.21.15.138:0/1406994601 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f19d409d840 0x7f19d4140980 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 >> 172.21.15.138:0/1406994601 conn(0x7f19d40aba10 msgr2=0x7f19d40a0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:33.903 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 shutdown_connections 2024-09-11T05:26:33.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 -- 172.21.15.138:0/1406994601 wait complete. 2024-09-11T05:26:33.904 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:33.903+0000 7f19da23d640 1 librados: shutdown 2024-09-11T05:26:33.905 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.905 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408700375}) 2024-09-11T05:26:33.905 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916065684}) 2024-09-11T05:26:33.905 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.905 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408700375}) 2024-09-11T05:26:33.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.903+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1853930363}) 2024-09-11T05:26:33.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126718994}) 2024-09-11T05:26:33.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1010681326}) 2024-09-11T05:26:33.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126718994}) 2024-09-11T05:26:33.906 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1069426200}) 2024-09-11T05:26:33.907 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.907 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844554440}) 2024-09-11T05:26:33.907 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3613078368}) 2024-09-11T05:26:33.907 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.907 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844554440}) 2024-09-11T05:26:33.907 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.907+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=695984308}) 2024-09-11T05:26:33.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332991258}) 2024-09-11T05:26:33.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1127790195}) 2024-09-11T05:26:33.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332991258}) 2024-09-11T05:26:33.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=411770286}) 2024-09-11T05:26:33.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549314337}) 2024-09-11T05:26:33.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=528318671}) 2024-09-11T05:26:33.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549314337}) 2024-09-11T05:26:33.910 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2006775598}) 2024-09-11T05:26:33.912 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.912 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656552852}) 2024-09-11T05:26:33.912 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1561365808}) 2024-09-11T05:26:33.913 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.913 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656552852}) 2024-09-11T05:26:33.913 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278139284}) 2024-09-11T05:26:33.914 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.914 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466466297}) 2024-09-11T05:26:33.914 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2664298156}) 2024-09-11T05:26:33.914 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.914 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466466297}) 2024-09-11T05:26:33.914 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.915+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2123509908}) 2024-09-11T05:26:33.916 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.916 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255093588}) 2024-09-11T05:26:33.916 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716036761}) 2024-09-11T05:26:33.916 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.916 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255093588}) 2024-09-11T05:26:33.916 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439510672}) 2024-09-11T05:26:33.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T05:26:33.918 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.918 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485446971}) 2024-09-11T05:26:33.918 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135593843}) 2024-09-11T05:26:33.918 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.918 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485446971}) 2024-09-11T05:26:33.918 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.919+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518373912}) 2024-09-11T05:26:33.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3412341246}) 2024-09-11T05:26:33.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2511149558}) 2024-09-11T05:26:33.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.919 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3412341246}) 2024-09-11T05:26:33.920 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952604996}) 2024-09-11T05:26:33.921 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.921 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419930210}) 2024-09-11T05:26:33.921 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282846062}) 2024-09-11T05:26:33.921 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.922 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419930210}) 2024-09-11T05:26:33.922 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.923+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2361598561}) 2024-09-11T05:26:33.923 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.923 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884490695}) 2024-09-11T05:26:33.923 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266920637}) 2024-09-11T05:26:33.923 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.924 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884490695}) 2024-09-11T05:26:33.924 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1842757838}) 2024-09-11T05:26:33.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670832297}) 2024-09-11T05:26:33.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875469405}) 2024-09-11T05:26:33.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670832297}) 2024-09-11T05:26:33.925 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.927+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377420906}) 2024-09-11T05:26:33.926 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578660049}) 2024-09-11T05:26:33.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2798025156}) 2024-09-11T05:26:33.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578660049}) 2024-09-11T05:26:33.927 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415843997}) 2024-09-11T05:26:33.928 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.928 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744434825}) 2024-09-11T05:26:33.928 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2245254942}) 2024-09-11T05:26:33.929 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.929 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744434825}) 2024-09-11T05:26:33.929 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615657}) 2024-09-11T05:26:33.930 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.930 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338033165}) 2024-09-11T05:26:33.930 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1534214309}) 2024-09-11T05:26:33.930 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.930 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338033165}) 2024-09-11T05:26:33.931 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.931+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525430484}) 2024-09-11T05:26:33.932 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.932 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9534372}) 2024-09-11T05:26:33.932 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1558802620}) 2024-09-11T05:26:33.932 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.932 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9534372}) 2024-09-11T05:26:33.932 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259017317}) 2024-09-11T05:26:33.934 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.934 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276128273}) 2024-09-11T05:26:33.934 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1262043556}) 2024-09-11T05:26:33.935 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.935 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276128273}) 2024-09-11T05:26:33.935 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.935+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3991441366}) 2024-09-11T05:26:33.936 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.936 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536494237}) 2024-09-11T05:26:33.936 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612904664}) 2024-09-11T05:26:33.936 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.936 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536494237}) 2024-09-11T05:26:33.936 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180964444}) 2024-09-11T05:26:33.938 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.938 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724125246}) 2024-09-11T05:26:33.938 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4026713696}) 2024-09-11T05:26:33.938 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.938 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724125246}) 2024-09-11T05:26:33.938 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.939+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1550187718}) 2024-09-11T05:26:33.939 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.940 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674119660}) 2024-09-11T05:26:33.940 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2655023820}) 2024-09-11T05:26:33.940 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.940 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674119660}) 2024-09-11T05:26:33.940 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907188910}) 2024-09-11T05:26:33.941 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.941 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083672469}) 2024-09-11T05:26:33.941 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135722832}) 2024-09-11T05:26:33.942 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.942 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083672469}) 2024-09-11T05:26:33.942 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.943+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256780954}) 2024-09-11T05:26:33.943 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.943 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407097946}) 2024-09-11T05:26:33.943 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878802122}) 2024-09-11T05:26:33.943 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.943 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407097946}) 2024-09-11T05:26:33.943 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3248118210}) 2024-09-11T05:26:33.945 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.945 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1932297029}) 2024-09-11T05:26:33.945 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1125599177}) 2024-09-11T05:26:33.945 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.945 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1932297029}) 2024-09-11T05:26:33.945 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.947+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3702302561}) 2024-09-11T05:26:33.947 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.947 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543932044}) 2024-09-11T05:26:33.947 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1296416457}) 2024-09-11T05:26:33.947 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.947 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543932044}) 2024-09-11T05:26:33.947 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931318740}) 2024-09-11T05:26:33.948 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680488640}) 2024-09-11T05:26:33.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=998475967}) 2024-09-11T05:26:33.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680488640}) 2024-09-11T05:26:33.949 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.951+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166338658}) 2024-09-11T05:26:33.950 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.951 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3935477605}) 2024-09-11T05:26:33.951 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2837719482}) 2024-09-11T05:26:33.951 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.951 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3935477605}) 2024-09-11T05:26:33.951 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065390758}) 2024-09-11T05:26:33.953 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.953 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349041038}) 2024-09-11T05:26:33.953 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643616547}) 2024-09-11T05:26:33.953 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.953 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349041038}) 2024-09-11T05:26:33.953 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.955+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=642841776}) 2024-09-11T05:26:33.955 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.955 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668976807}) 2024-09-11T05:26:33.955 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130023179}) 2024-09-11T05:26:33.955 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.955 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668976807}) 2024-09-11T05:26:33.955 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2463037922}) 2024-09-11T05:26:33.956 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.957 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121721808}) 2024-09-11T05:26:33.957 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454088142}) 2024-09-11T05:26:33.957 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.957 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121721808}) 2024-09-11T05:26:33.957 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945119660}) 2024-09-11T05:26:33.958 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.958 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939864396}) 2024-09-11T05:26:33.958 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2809035989}) 2024-09-11T05:26:33.958 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.958 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939864396}) 2024-09-11T05:26:33.958 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.959+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1378542200}) 2024-09-11T05:26:33.960 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.960 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896842190}) 2024-09-11T05:26:33.960 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3007318740}) 2024-09-11T05:26:33.960 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.960 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896842190}) 2024-09-11T05:26:33.960 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898142201}) 2024-09-11T05:26:33.962 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.962 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4148741414}) 2024-09-11T05:26:33.962 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1385430423}) 2024-09-11T05:26:33.962 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.962 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4148741414}) 2024-09-11T05:26:33.962 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.963+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126826010}) 2024-09-11T05:26:33.963 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.964 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270027892}) 2024-09-11T05:26:33.964 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283890355}) 2024-09-11T05:26:33.964 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.964 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270027892}) 2024-09-11T05:26:33.964 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3504909038}) 2024-09-11T05:26:33.966 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.966 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=243129430}) 2024-09-11T05:26:33.966 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=787165575}) 2024-09-11T05:26:33.966 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.966 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=243129430}) 2024-09-11T05:26:33.966 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.967+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176445664}) 2024-09-11T05:26:33.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057482483}) 2024-09-11T05:26:33.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989637343}) 2024-09-11T05:26:33.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057482483}) 2024-09-11T05:26:33.968 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080782202}) 2024-09-11T05:26:33.970 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.970 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930245927}) 2024-09-11T05:26:33.970 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838624141}) 2024-09-11T05:26:33.970 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.970 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930245927}) 2024-09-11T05:26:33.970 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.971+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4202626974}) 2024-09-11T05:26:33.972 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.972 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505301341}) 2024-09-11T05:26:33.972 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=51609029}) 2024-09-11T05:26:33.972 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.972 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505301341}) 2024-09-11T05:26:33.972 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2743930816}) 2024-09-11T05:26:33.974 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.974 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779178119}) 2024-09-11T05:26:33.974 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2092970349}) 2024-09-11T05:26:33.974 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.974 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779178119}) 2024-09-11T05:26:33.974 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.975+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1046493071}) 2024-09-11T05:26:33.976 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.976 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730290303}) 2024-09-11T05:26:33.976 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148653270}) 2024-09-11T05:26:33.976 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.976 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730290303}) 2024-09-11T05:26:33.976 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4181698122}) 2024-09-11T05:26:33.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390463339}) 2024-09-11T05:26:33.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020038115}) 2024-09-11T05:26:33.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390463339}) 2024-09-11T05:26:33.978 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.979+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491281443}) 2024-09-11T05:26:33.979 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.979 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065949316}) 2024-09-11T05:26:33.979 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958546829}) 2024-09-11T05:26:33.980 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.980 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065949316}) 2024-09-11T05:26:33.980 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3766302536}) 2024-09-11T05:26:33.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431675219}) 2024-09-11T05:26:33.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1833787563}) 2024-09-11T05:26:33.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.981 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431675219}) 2024-09-11T05:26:33.982 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.983+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3863339804}) 2024-09-11T05:26:33.983 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.983 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=111922301}) 2024-09-11T05:26:33.983 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4048543059}) 2024-09-11T05:26:33.983 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.983 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=111922301}) 2024-09-11T05:26:33.983 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1566973349}) 2024-09-11T05:26:33.985 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.985 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530253433}) 2024-09-11T05:26:33.985 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057628580}) 2024-09-11T05:26:33.985 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.985 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530253433}) 2024-09-11T05:26:33.985 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.987+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4270079319}) 2024-09-11T05:26:33.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051174827}) 2024-09-11T05:26:33.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3016317376}) 2024-09-11T05:26:33.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051174827}) 2024-09-11T05:26:33.987 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654227675}) 2024-09-11T05:26:33.989 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.989 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303190918}) 2024-09-11T05:26:33.989 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572062735}) 2024-09-11T05:26:33.989 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.989 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303190918}) 2024-09-11T05:26:33.989 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.991+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3801433948}) 2024-09-11T05:26:33.990 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.990 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277199775}) 2024-09-11T05:26:33.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=779629386}) 2024-09-11T05:26:33.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277199775}) 2024-09-11T05:26:33.991 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647590314}) 2024-09-11T05:26:33.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.992 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606250853}) 2024-09-11T05:26:33.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=363951440}) 2024-09-11T05:26:33.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606250853}) 2024-09-11T05:26:33.993 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139583570}) 2024-09-11T05:26:33.994 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.994 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386636445}) 2024-09-11T05:26:33.994 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=263937740}) 2024-09-11T05:26:33.995 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.995 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386636445}) 2024-09-11T05:26:33.995 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.995+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698305242}) 2024-09-11T05:26:33.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934663255}) 2024-09-11T05:26:33.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3238330511}) 2024-09-11T05:26:33.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934663255}) 2024-09-11T05:26:33.996 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=176122545}) 2024-09-11T05:26:33.998 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.998 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260372033}) 2024-09-11T05:26:33.998 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=625522700}) 2024-09-11T05:26:33.998 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.998 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4260372033}) 2024-09-11T05:26:33.998 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:33.999+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965758936}) 2024-09-11T05:26:33.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569102020}) 2024-09-11T05:26:33.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850307352}) 2024-09-11T05:26:33.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:33.999 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569102020}) 2024-09-11T05:26:34.000 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039232151}) 2024-09-11T05:26:34.001 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.001 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137948027}) 2024-09-11T05:26:34.001 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2697315578}) 2024-09-11T05:26:34.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137948027}) 2024-09-11T05:26:34.002 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.003+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440342730}) 2024-09-11T05:26:34.003 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.003 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870267110}) 2024-09-11T05:26:34.003 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1102141364}) 2024-09-11T05:26:34.003 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.003 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870267110}) 2024-09-11T05:26:34.003 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=672674866}) 2024-09-11T05:26:34.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155110787}) 2024-09-11T05:26:34.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3300688116}) 2024-09-11T05:26:34.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155110787}) 2024-09-11T05:26:34.005 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.007+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=996903622}) 2024-09-11T05:26:34.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527670138}) 2024-09-11T05:26:34.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478164374}) 2024-09-11T05:26:34.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527670138}) 2024-09-11T05:26:34.007 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737838800}) 2024-09-11T05:26:34.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1567379816}) 2024-09-11T05:26:34.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=826563198}) 2024-09-11T05:26:34.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1567379816}) 2024-09-11T05:26:34.009 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.011+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522299191}) 2024-09-11T05:26:34.010 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.010 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349092743}) 2024-09-11T05:26:34.011 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755654551}) 2024-09-11T05:26:34.011 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.011 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349092743}) 2024-09-11T05:26:34.011 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1844139161}) 2024-09-11T05:26:34.012 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.013 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090137853}) 2024-09-11T05:26:34.013 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772447406}) 2024-09-11T05:26:34.013 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.013 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090137853}) 2024-09-11T05:26:34.013 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285281365}) 2024-09-11T05:26:34.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.015+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204344671}) 2024-09-11T05:26:34.014 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1560998901}) 2024-09-11T05:26:34.015 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.015 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204344671}) 2024-09-11T05:26:34.015 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342207648}) 2024-09-11T05:26:34.017 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.017 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638803716}) 2024-09-11T05:26:34.017 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1145827401}) 2024-09-11T05:26:34.017 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.017 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638803716}) 2024-09-11T05:26:34.017 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477268246}) 2024-09-11T05:26:34.018 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.018 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.019+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580415106}) 2024-09-11T05:26:34.019 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=537532121}) 2024-09-11T05:26:34.019 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.019 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580415106}) 2024-09-11T05:26:34.019 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3819102212}) 2024-09-11T05:26:34.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394484506}) 2024-09-11T05:26:34.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=587962379}) 2024-09-11T05:26:34.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.020 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394484506}) 2024-09-11T05:26:34.021 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.023+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3614307784}) 2024-09-11T05:26:34.022 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=89,mdsmap=3,mgr=7,mgr_command_descs=1} crc {logm=312256687,mdsmap=1754867906,mgr=3841245533,mgr_command_descs=2308730518}) 2024-09-11T05:26:34.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=2507952232,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=3742250831}) 2024-09-11T05:26:34.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=89,mdsmap=3,mgr=7,mgr_command_descs=1} crc {logm=312256687,mdsmap=1754867906,mgr=3841245533,mgr_command_descs=2308730518}) 2024-09-11T05:26:34.023 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1088057713,mdsmap=1754867906,mgr=881679835}) 2024-09-11T05:26:34.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=2146688854,config=692619927,health=3244156202,kv=310889196,logm=751855312}) 2024-09-11T05:26:34.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=2146688854,config=692619927,health=3244156202,kv=310889196,logm=1110016109}) 2024-09-11T05:26:34.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=2146688854,config=692619927,health=3244156202,kv=310889196,logm=751855312}) 2024-09-11T05:26:34.024 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=3630084024,config=692619927,health=1983435026,kv=310889196,logm=167693029}) 2024-09-11T05:26:34.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381604024}) 2024-09-11T05:26:34.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2202349469}) 2024-09-11T05:26:34.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381604024}) 2024-09-11T05:26:34.026 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.027+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889525787}) 2024-09-11T05:26:34.028 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.028 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021131301}) 2024-09-11T05:26:34.028 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=425077615}) 2024-09-11T05:26:34.028 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.028 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021131301}) 2024-09-11T05:26:34.028 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352413618}) 2024-09-11T05:26:34.029 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.030 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889461952}) 2024-09-11T05:26:34.030 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3111534878}) 2024-09-11T05:26:34.030 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.030 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889461952}) 2024-09-11T05:26:34.030 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.031+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3075302967}) 2024-09-11T05:26:34.031 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100997802}) 2024-09-11T05:26:34.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2060332986}) 2024-09-11T05:26:34.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100997802}) 2024-09-11T05:26:34.032 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1560985114}) 2024-09-11T05:26:34.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1756800622}) 2024-09-11T05:26:34.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3264554563}) 2024-09-11T05:26:34.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.033 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1756800622}) 2024-09-11T05:26:34.034 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.035+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278496304}) 2024-09-11T05:26:34.035 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.035 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545271024}) 2024-09-11T05:26:34.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442347716}) 2024-09-11T05:26:34.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545271024}) 2024-09-11T05:26:34.036 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339842764}) 2024-09-11T05:26:34.037 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.038 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134028209}) 2024-09-11T05:26:34.038 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174233344}) 2024-09-11T05:26:34.038 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.038 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134028209}) 2024-09-11T05:26:34.038 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.039+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2963025040}) 2024-09-11T05:26:34.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125794061}) 2024-09-11T05:26:34.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2319741524}) 2024-09-11T05:26:34.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125794061}) 2024-09-11T05:26:34.040 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3778764501}) 2024-09-11T05:26:34.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715209456}) 2024-09-11T05:26:34.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544732942}) 2024-09-11T05:26:34.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715209456}) 2024-09-11T05:26:34.042 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.043+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1767177192}) 2024-09-11T05:26:34.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93480155}) 2024-09-11T05:26:34.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4171555108}) 2024-09-11T05:26:34.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93480155}) 2024-09-11T05:26:34.044 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967996980}) 2024-09-11T05:26:34.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592577855}) 2024-09-11T05:26:34.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1601797267}) 2024-09-11T05:26:34.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592577855}) 2024-09-11T05:26:34.046 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.047+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733364934}) 2024-09-11T05:26:34.048 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744128037}) 2024-09-11T05:26:34.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3624594875}) 2024-09-11T05:26:34.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744128037}) 2024-09-11T05:26:34.049 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1855050785}) 2024-09-11T05:26:34.051 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.051 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129451185}) 2024-09-11T05:26:34.051 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1125673229}) 2024-09-11T05:26:34.051 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.051 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129451185}) 2024-09-11T05:26:34.051 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.051+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2921053001}) 2024-09-11T05:26:34.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2375126570}) 2024-09-11T05:26:34.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091559750}) 2024-09-11T05:26:34.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2375126570}) 2024-09-11T05:26:34.052 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1424039295}) 2024-09-11T05:26:34.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954712616}) 2024-09-11T05:26:34.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551583526}) 2024-09-11T05:26:34.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=954712616}) 2024-09-11T05:26:34.054 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.055+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3669893833}) 2024-09-11T05:26:34.056 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.056 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888268459}) 2024-09-11T05:26:34.056 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162699663}) 2024-09-11T05:26:34.056 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.056 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2888268459}) 2024-09-11T05:26:34.057 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594776059}) 2024-09-11T05:26:34.058 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.058 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102594960}) 2024-09-11T05:26:34.058 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2766110563}) 2024-09-11T05:26:34.058 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.058 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102594960}) 2024-09-11T05:26:34.059 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.059+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=209134914}) 2024-09-11T05:26:34.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019288370}) 2024-09-11T05:26:34.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3260626286}) 2024-09-11T05:26:34.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019288370}) 2024-09-11T05:26:34.060 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606277421}) 2024-09-11T05:26:34.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959342194}) 2024-09-11T05:26:34.062 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759896605}) 2024-09-11T05:26:34.063 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.063 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3959342194}) 2024-09-11T05:26:34.063 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.063+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=837897013}) 2024-09-11T05:26:34.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046216189}) 2024-09-11T05:26:34.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=948716488}) 2024-09-11T05:26:34.064 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.065 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046216189}) 2024-09-11T05:26:34.065 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=276115124}) 2024-09-11T05:26:34.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101531013}) 2024-09-11T05:26:34.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904774713}) 2024-09-11T05:26:34.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101531013}) 2024-09-11T05:26:34.066 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.067+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=392520869}) 2024-09-11T05:26:34.074 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.075+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.074 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.075+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1636762140}) 2024-09-11T05:26:34.074 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.075+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1579140791}) 2024-09-11T05:26:34.075 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.075+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.075 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.075+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1636762140}) 2024-09-11T05:26:34.075 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.075+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715896856}) 2024-09-11T05:26:34.076 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.079+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.079+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181766101}) 2024-09-11T05:26:34.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.079+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3312656044}) 2024-09-11T05:26:34.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.079+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.079+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181766101}) 2024-09-11T05:26:34.077 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.079+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721094637}) 2024-09-11T05:26:34.078 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.078 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141295676}) 2024-09-11T05:26:34.078 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2466860909}) 2024-09-11T05:26:34.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141295676}) 2024-09-11T05:26:34.079 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2272105804}) 2024-09-11T05:26:34.080 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2532044833}) 2024-09-11T05:26:34.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693526443}) 2024-09-11T05:26:34.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2532044833}) 2024-09-11T05:26:34.081 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.083+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2863169047}) 2024-09-11T05:26:34.083 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.083 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1298376343}) 2024-09-11T05:26:34.083 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=534562741}) 2024-09-11T05:26:34.083 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.083 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1298376343}) 2024-09-11T05:26:34.083 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001594674}) 2024-09-11T05:26:34.084 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.085 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353889941}) 2024-09-11T05:26:34.085 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3378140768}) 2024-09-11T05:26:34.085 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.085 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353889941}) 2024-09-11T05:26:34.085 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.087+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2498862670}) 2024-09-11T05:26:34.412 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.412 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090894549}) 2024-09-11T05:26:34.412 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142471411}) 2024-09-11T05:26:34.412 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.412 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090894549}) 2024-09-11T05:26:34.412 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1317255750}) 2024-09-11T05:26:34.414 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.414 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316405364}) 2024-09-11T05:26:34.414 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1778971116}) 2024-09-11T05:26:34.414 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.414 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316405364}) 2024-09-11T05:26:34.414 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.415+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297921595}) 2024-09-11T05:26:34.416 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.419+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.416 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.419+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962371278}) 2024-09-11T05:26:34.416 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.419+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3090442894}) 2024-09-11T05:26:34.416 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.419+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.417 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.419+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962371278}) 2024-09-11T05:26:34.417 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.419+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3617134444}) 2024-09-11T05:26:34.418 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.419+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.418 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644513651}) 2024-09-11T05:26:34.419 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2137517227}) 2024-09-11T05:26:34.419 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.419 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644513651}) 2024-09-11T05:26:34.419 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251254849}) 2024-09-11T05:26:34.421 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.421 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094427971}) 2024-09-11T05:26:34.421 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282272443}) 2024-09-11T05:26:34.421 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.421 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094427971}) 2024-09-11T05:26:34.422 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.423+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3857181295}) 2024-09-11T05:26:34.423 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.423 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250648180}) 2024-09-11T05:26:34.423 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2320918331}) 2024-09-11T05:26:34.423 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.423 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250648180}) 2024-09-11T05:26:34.423 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1940176635}) 2024-09-11T05:26:34.425 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.425 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741878925}) 2024-09-11T05:26:34.425 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2682731896}) 2024-09-11T05:26:34.425 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.425 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741878925}) 2024-09-11T05:26:34.425 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.427+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=277906675}) 2024-09-11T05:26:34.427 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.427 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968907935}) 2024-09-11T05:26:34.427 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693255015}) 2024-09-11T05:26:34.427 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.427 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968907935}) 2024-09-11T05:26:34.427 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192982609}) 2024-09-11T05:26:34.429 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.430 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078892415}) 2024-09-11T05:26:34.430 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1980112382}) 2024-09-11T05:26:34.430 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.430 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078892415}) 2024-09-11T05:26:34.430 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.431+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511090146}) 2024-09-11T05:26:34.432 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.432 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635518448}) 2024-09-11T05:26:34.432 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280236992}) 2024-09-11T05:26:34.432 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.432 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635518448}) 2024-09-11T05:26:34.432 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482959954}) 2024-09-11T05:26:34.434 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.434 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649033855}) 2024-09-11T05:26:34.434 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1296848686}) 2024-09-11T05:26:34.434 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.434 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649033855}) 2024-09-11T05:26:34.434 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.435+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1374749501}) 2024-09-11T05:26:34.436 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.436 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004246297}) 2024-09-11T05:26:34.436 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4134612471}) 2024-09-11T05:26:34.436 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.436 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004246297}) 2024-09-11T05:26:34.436 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1561843137}) 2024-09-11T05:26:34.439 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.439 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399122611}) 2024-09-11T05:26:34.439 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2066671172}) 2024-09-11T05:26:34.439 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.439 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399122611}) 2024-09-11T05:26:34.439 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.439+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2552717378}) 2024-09-11T05:26:34.440 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.440 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656892970}) 2024-09-11T05:26:34.440 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1415198983}) 2024-09-11T05:26:34.440 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.440 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656892970}) 2024-09-11T05:26:34.441 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3658772231}) 2024-09-11T05:26:34.442 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.442 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481453043}) 2024-09-11T05:26:34.442 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=93836365}) 2024-09-11T05:26:34.442 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.443 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481453043}) 2024-09-11T05:26:34.443 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.443+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2354231945}) 2024-09-11T05:26:34.444 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.447+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.444 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.447+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493374024}) 2024-09-11T05:26:34.445 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.447+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=740572204}) 2024-09-11T05:26:34.445 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.447+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.445 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.447+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493374024}) 2024-09-11T05:26:34.445 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.447+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1944226190}) 2024-09-11T05:26:34.446 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.447 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408457613}) 2024-09-11T05:26:34.447 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2648849763}) 2024-09-11T05:26:34.447 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.447 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408457613}) 2024-09-11T05:26:34.447 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1878332742}) 2024-09-11T05:26:34.449 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.449 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738189012}) 2024-09-11T05:26:34.449 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089254893}) 2024-09-11T05:26:34.449 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.449 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738189012}) 2024-09-11T05:26:34.449 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.451+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=876263775}) 2024-09-11T05:26:34.451 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.451 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991930739}) 2024-09-11T05:26:34.451 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=171386112}) 2024-09-11T05:26:34.451 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.451 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991930739}) 2024-09-11T05:26:34.451 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3586582554}) 2024-09-11T05:26:34.453 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.453 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280522135}) 2024-09-11T05:26:34.453 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2029756991}) 2024-09-11T05:26:34.453 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.453 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280522135}) 2024-09-11T05:26:34.454 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.455+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1828660365}) 2024-09-11T05:26:34.455 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.455 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1055022057}) 2024-09-11T05:26:34.455 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427445552}) 2024-09-11T05:26:34.455 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.456 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1055022057}) 2024-09-11T05:26:34.456 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3336997845}) 2024-09-11T05:26:34.457 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.457 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869335411}) 2024-09-11T05:26:34.458 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1369410424}) 2024-09-11T05:26:34.458 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.458 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869335411}) 2024-09-11T05:26:34.458 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.459+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=250129968}) 2024-09-11T05:26:34.460 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.460 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89048267}) 2024-09-11T05:26:34.460 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=991053664}) 2024-09-11T05:26:34.460 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.460 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89048267}) 2024-09-11T05:26:34.460 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145113075}) 2024-09-11T05:26:34.462 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.462 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677430080}) 2024-09-11T05:26:34.462 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2302644348}) 2024-09-11T05:26:34.462 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.462 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677430080}) 2024-09-11T05:26:34.462 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.463+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1426389050}) 2024-09-11T05:26:34.464 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.467+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.464 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.467+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090425089}) 2024-09-11T05:26:34.464 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.467+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875556474}) 2024-09-11T05:26:34.464 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.467+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.464 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.467+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090425089}) 2024-09-11T05:26:34.464 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.467+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369370718}) 2024-09-11T05:26:34.474 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.475+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.474 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.475+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478156177}) 2024-09-11T05:26:34.474 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.475+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3632624320}) 2024-09-11T05:26:34.474 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.475+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.474 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.475+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478156177}) 2024-09-11T05:26:34.474 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.475+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232261621}) 2024-09-11T05:26:34.668 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.668 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329898702}) 2024-09-11T05:26:34.669 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3619947964}) 2024-09-11T05:26:34.669 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.669 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329898702}) 2024-09-11T05:26:34.669 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1214457520}) 2024-09-11T05:26:34.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.815+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.815+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862357606}) 2024-09-11T05:26:34.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.815+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057859978}) 2024-09-11T05:26:34.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.815+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.815+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862357606}) 2024-09-11T05:26:34.813 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.815+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945367628}) 2024-09-11T05:26:34.871 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.875+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.872 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.875+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410647410}) 2024-09-11T05:26:34.872 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.875+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902265608}) 2024-09-11T05:26:34.872 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.875+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:34.872 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.875+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410647410}) 2024-09-11T05:26:34.872 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:34.875+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1655067143}) 2024-09-11T05:26:34.919 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-11T05:26:34.919 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T05:26:35.044 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.047+0000 7f04059ce640 10 monclient: get_monmap_and_config 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.047+0000 7f04059ce640 10 monclient: build_initial_monmap 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.047+0000 7f04059ce640 10 monclient: monmap: 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:35.051373+0000 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:35.051373+0000 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:35.045 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:35.046 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:35.046 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.047+0000 7f04059ce640 1 Processor -- start 2024-09-11T05:26:35.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.047+0000 7f04059ce640 1 -- start start 2024-09-11T05:26:35.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.047+0000 7f04059ce640 10 monclient: init 2024-09-11T05:26:35.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:35.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-11T05:26:35.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f040006e0e0 0x7f040009e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient(hunting): picked mon.noname-a con 0x7f040006e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:35.048 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f040009e960 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:35.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f0400135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient(hunting): picked mon.noname-b con 0x7f040009ed70 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:35.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f040009f540 con 0x7f040009e960 2024-09-11T05:26:35.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f040006e4c0 con 0x7f040006e0e0 2024-09-11T05:26:35.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f040009f820 con 0x7f040009ed70 2024-09-11T05:26:35.049 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f04059ce640 10 monclient(hunting): authenticate will time out at 2078.488235s 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f0400135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f040006e0e0 0x7f040009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f040006e0e0 0x7f040009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40830/0 (socket says 172.21.15.138:40830) 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f0400135e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55586/0 (socket says 172.21.15.138:55586) 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 1 -- 172.21.15.138:0/1718694002 learned_addr learned my addr 172.21.15.138:0/1718694002 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient(hunting): get_auth_request con 0x7f040006e0e0 auth_method 0 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 10 monclient(hunting): get_auth_request con 0x7f040009ed70 auth_method 0 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.050 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient(hunting): handle_auth_done global_id 114321 payload 274 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 1 -- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 msgr2=0x7f0400135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 1 --2- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f0400135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 1 -- 172.21.15.138:0/1718694002 >> v1:172.21.15.190:6789/0 conn(0x7f040009e960 legacy=0x7f04001356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient: _finish_hunting 0 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 1 monclient: found mon.noname-a 2024-09-11T05:26:35.051 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:35.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03feffd640 1 -- 172.21.15.138:0/1718694002 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040009fb00 con 0x7f040006e0e0 2024-09-11T05:26:35.052 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.051+0000 7f03ff7fe640 1 --2- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f0400135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:35.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03feffd640 1 -- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040006e0e0 msgr2=0x7f040009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:35.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03feffd640 1 --2- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040006e0e0 0x7f040009e420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03f00098d0 tx=0x7f03f001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337e47c88140671 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 1 -- 172.21.15.138:0/1718694002 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f00b0070 con 0x7f040006e0e0 2024-09-11T05:26:35.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:35.056 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: dump: 2024-09-11T05:26:35.056 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: _finish_auth 0 2024-09-11T05:26:35.057 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 5 monclient: authenticate success, global_id 114321 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 1 -- 172.21.15.138:0/1718694002 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f00a4e50 con 0x7f040006e0e0 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 1 -- 172.21.15.138:0/1718694002 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f00ab820 con 0x7f040006e0e0 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 4 set_mon_vals no callback set 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03fe7fc640 10 monclient: dump: 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.058 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient: shutdown 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040006e0e0 msgr2=0x7f040009e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 --2- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040006e0e0 0x7f040009e420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03f00098d0 tx=0x7f03f001c400 comp rx=0 tx=0).stop 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- 172.21.15.138:0/1718694002 shutdown_connections 2024-09-11T05:26:35.059 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 --2- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f0400135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 --2- 172.21.15.138:0/1718694002 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040006e0e0 0x7f040009e420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- 172.21.15.138:0/1718694002 >> 172.21.15.138:0/1718694002 conn(0x7f0400063640 msgr2=0x7f0400063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- 172.21.15.138:0/1718694002 shutdown_connections 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- 172.21.15.138:0/1718694002 wait complete. 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient: build_initial_monmap 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient: monmap: 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:35.060 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 librados: starting msgr at 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 librados: starting objecter 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 Processor -- start 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- start start 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 librados: setting wanted keys 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 librados: calling monclient init 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient: init 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f040006e0e0 0x7f040012e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.061 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient(hunting): picked mon.noname-b con 0x7f040006e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040009e960 0x7f040012ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient(hunting): picked mon.noname-a con 0x7f040009e960 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f040013ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient(hunting): picked mon.noname-c con 0x7f040009ed70 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f04001364c0 con 0x7f040009e960 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0400136340 con 0x7f040006e0e0 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f04000a29c0 con 0x7f040009ed70 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:35.062 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f04059ce640 10 monclient(hunting): authenticate will time out at 2078.495460s 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040009e960 0x7f040012ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03ff7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f040013ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040009e960 0x7f040012ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40834/0 (socket says 172.21.15.138:40834) 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 -- 172.21.15.138:0/1770004432 learned_addr learned my addr 172.21.15.138:0/1770004432 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient(hunting): get_auth_request con 0x7f040009e960 auth_method 0 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03ff7fe640 10 monclient(hunting): get_auth_request con 0x7f040009ed70 auth_method 0 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.063 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03feffd640 1 --2- 172.21.15.138:0/1770004432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f040006e0e0 0x7f040012e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03feffd640 10 monclient(hunting): get_auth_request con 0x7f040006e0e0 auth_method 0 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient(hunting): handle_auth_done global_id 114324 payload 824 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 -- 172.21.15.138:0/1770004432 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 msgr2=0x7f040013ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 --2- 172.21.15.138:0/1770004432 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f040013ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 -- 172.21.15.138:0/1770004432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f040006e0e0 msgr2=0x7f040012e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 --2- 172.21.15.138:0/1770004432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f040006e0e0 0x7f040012e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient: _finish_hunting 0 2024-09-11T05:26:35.064 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 monclient: found mon.noname-a 2024-09-11T05:26:35.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:35.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.065 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.055+0000 7f03f67fc640 1 -- 172.21.15.138:0/1770004432 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040009d500 con 0x7f040009e960 2024-09-11T05:26:35.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03f67fc640 1 -- 172.21.15.138:0/1770004432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040009e960 msgr2=0x7f040012ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:35.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03f67fc640 1 --2- 172.21.15.138:0/1770004432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040009e960 0x7f040012ebc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f03e800c480 tx=0x7f03e800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 client.?.objecter ms_handle_connect 0x7f040009e960 2024-09-11T05:26:35.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:35.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 1 -- 172.21.15.138:0/1770004432 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8012a40 con 0x7f040009e960 2024-09-11T05:26:35.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.067 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: dump: 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: _finish_auth 0 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: _check_auth_tickets 2024-09-11T05:26:35.068 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 1 -- 172.21.15.138:0/1770004432 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e8017070 con 0x7f040009e960 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 5 monclient: authenticate success, global_id 114324 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 10 monclient: _renew_subs 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0400073f10 con 0x7f040009e960 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 1 -- 172.21.15.138:0/1770004432 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8012e40 con 0x7f040009e960 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 monclient: dump: 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.069 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fd7fa640 4 set_mon_vals no callback set 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 10 client.114324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 10 monclient: _renew_subs 2024-09-11T05:26:35.070 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0400073630 con 0x7f040009e960 2024-09-11T05:26:35.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 1 librados: init done 2024-09-11T05:26:35.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.071 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04000a2e20 con 0x7f040009e960 2024-09-11T05:26:35.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 1 -- 172.21.15.138:0/1770004432 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f03e801c460 con 0x7f040009e960 2024-09-11T05:26:35.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 1 --2- 172.21.15.138:0/1770004432 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03e003cf20 0x7f03e003f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 1 -- 172.21.15.138:0/1770004432 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f03e804f500 con 0x7f040009e960 2024-09-11T05:26:35.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 10 client.114324.objecter ms_dispatch 0x7f0400000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:35.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 3 client.114324.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:35.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 3 client.114324.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:35.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.059+0000 7f03fcff9640 20 client.114324.objecter dump_active .. 0 homeless 2024-09-11T05:26:35.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.063+0000 7f03feffd640 1 --2- 172.21.15.138:0/1770004432 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03e003cf20 0x7f03e003f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.063+0000 7f03feffd640 10 monclient: get_auth_request con 0x7f03e003cf20 auth_method 0 2024-09-11T05:26:35.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.063+0000 7f03feffd640 1 --2- 172.21.15.138:0/1770004432 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03e003cf20 0x7f03e003f3e0 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7f03f00098a0 tx=0x7f03f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.063+0000 7f03fcff9640 10 client.114324.objecter ms_handle_connect 0x7f03e003cf20 2024-09-11T05:26:35.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.067+0000 7f03fcff9640 1 -- 172.21.15.138:0/1770004432 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f03e801ac60 con 0x7f040009e960 2024-09-11T05:26:35.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.067+0000 7f03fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.067+0000 7f03fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:35.269 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.271+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.269 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.271+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528820036}) 2024-09-11T05:26:35.269 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.271+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010623332}) 2024-09-11T05:26:35.269 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.271+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.269 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.271+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528820036}) 2024-09-11T05:26:35.269 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.271+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2950824547}) 2024-09-11T05:26:35.271 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.275+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.271 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.275+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2889513991}) 2024-09-11T05:26:35.271 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.275+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4082314568}) 2024-09-11T05:26:35.271 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.275+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.271 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.275+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2889513991}) 2024-09-11T05:26:35.271 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.275+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246613074}) 2024-09-11T05:26:35.283 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.283 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654315393}) 2024-09-11T05:26:35.283 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526183621}) 2024-09-11T05:26:35.283 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.283 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654315393}) 2024-09-11T05:26:35.283 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1597568917}) 2024-09-11T05:26:35.285 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.285 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117463527}) 2024-09-11T05:26:35.285 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1529296541}) 2024-09-11T05:26:35.285 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.285 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117463527}) 2024-09-11T05:26:35.285 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.287+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322823513}) 2024-09-11T05:26:35.286 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.286 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172988458}) 2024-09-11T05:26:35.286 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163176501}) 2024-09-11T05:26:35.287 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.287 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172988458}) 2024-09-11T05:26:35.287 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700948865}) 2024-09-11T05:26:35.288 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.288 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199584608}) 2024-09-11T05:26:35.288 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1276545909}) 2024-09-11T05:26:35.288 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.288 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199584608}) 2024-09-11T05:26:35.288 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283432791}) 2024-09-11T05:26:35.290 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.290 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2473117603}) 2024-09-11T05:26:35.290 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4287546871}) 2024-09-11T05:26:35.290 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.290 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2473117603}) 2024-09-11T05:26:35.290 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.291+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=489068333}) 2024-09-11T05:26:35.292 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.295+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.292 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.295+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416688747}) 2024-09-11T05:26:35.292 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.295+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=211747434}) 2024-09-11T05:26:35.292 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.295+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.292 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.295+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416688747}) 2024-09-11T05:26:35.292 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.295+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=626717778}) 2024-09-11T05:26:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.307+0000 7f04059ce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T05:26:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.307+0000 7f04059ce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T05:26:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.307+0000 7f04059ce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.305 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.307+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f040009dea0 con 0x7f040009e960 2024-09-11T05:26:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.307+0000 7f03fcff9640 1 -- 172.21.15.138:0/1770004432 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f03e801c960 con 0x7f040009e960 2024-09-11T05:26:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.307+0000 7f03fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T05:26:35.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.307+0000 7f03fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:26:35.306 INFO:teuthology.orchestra.run.smithi138.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","modified":"2024-09-11T05:06:55.201195Z","created":"2024-09-11T05:06:55.201195Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3301","nonce":0},{"type":"v1","addr":"172.21.15.138:6790","nonce":0}]},"addr":"172.21.15.138:6790/0","public_addr":"172.21.15.138:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T05:26:35.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 10 librados: watch_flush enter 2024-09-11T05:26:35.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 10 librados: watch_flush exit 2024-09-11T05:26:35.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 20 client.114324.objecter shutdown clearing up homeless session... 2024-09-11T05:26:35.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 10 client.114324.objecter successfully canceled tick 2024-09-11T05:26:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03e003cf20 msgr2=0x7f03e003f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 --2- 172.21.15.138:0/1770004432 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03e003cf20 0x7f03e003f3e0 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7f03f00098a0 tx=0x7f03f0008040 comp rx=0 tx=0).stop 2024-09-11T05:26:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 10 monclient: shutdown 2024-09-11T05:26:35.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040009e960 msgr2=0x7f040012ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 --2- 172.21.15.138:0/1770004432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040009e960 0x7f040012ebc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f03e800c480 tx=0x7f03e800c940 comp rx=0 tx=0).stop 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 shutdown_connections 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 --2- 172.21.15.138:0/1770004432 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03e003cf20 0x7f03e003f3e0 unknown :-1 s=CLOSED pgs=3518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 --2- 172.21.15.138:0/1770004432 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f040009ed70 0x7f040013ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 --2- 172.21.15.138:0/1770004432 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f040009e960 0x7f040012ebc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 --2- 172.21.15.138:0/1770004432 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f040006e0e0 0x7f040012e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 >> 172.21.15.138:0/1770004432 conn(0x7f0400063640 msgr2=0x7f04000a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 shutdown_connections 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 -- 172.21.15.138:0/1770004432 wait complete. 2024-09-11T05:26:35.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.311+0000 7f04059ce640 1 librados: shutdown 2024-09-11T05:26:35.325 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-11T05:26:35.325 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T05:26:35.334 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T05:26:35.334 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:26:35.457 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.459+0000 7efcd73b1640 10 monclient: get_monmap_and_config 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.459+0000 7efcd73b1640 10 monclient: build_initial_monmap 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.459+0000 7efcd73b1640 10 monclient: monmap: 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:35.464374+0000 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:35.464374+0000 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:35.458 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:35.459 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:35.459 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.459+0000 7efcd73b1640 1 Processor -- start 2024-09-11T05:26:35.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.459+0000 7efcd73b1640 1 -- start start 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.459+0000 7efcd73b1640 10 monclient: init 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efcd006e0e0 0x7efcd009e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient(hunting): picked mon.noname-c con 0x7efcd006e0e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efcd009e960 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7efcd009ed70 0x7efcd0135e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient(hunting): picked mon.noname-a con 0x7efcd009ed70 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcd006e4c0 con 0x7efcd009e960 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7efcd009f820 con 0x7efcd009ed70 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efcd009f540 con 0x7efcd006e0e0 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd73b1640 10 monclient(hunting): authenticate will time out at 2078.900447s 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7efcd009ed70 0x7efcd0135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd494c640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7efcd009e960 0x7efcd01356f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50010/0 (socket says 172.21.15.138:50010) 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd494c640 1 -- 172.21.15.138:0/2811668793 learned_addr learned my addr 172.21.15.138:0/2811668793 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient(hunting): get_auth_request con 0x7efcd009ed70 auth_method 0 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd514d640 1 --2- 172.21.15.138:0/2811668793 >> v2:172.21.15.190:3300/0 conn(0x7efcd006e0e0 0x7efcd009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.462 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient(hunting): handle_auth_done global_id 114327 payload 274 2024-09-11T05:26:35.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 1 -- 172.21.15.138:0/2811668793 >> v2:172.21.15.190:3300/0 conn(0x7efcd006e0e0 msgr2=0x7efcd009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:35.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 1 --2- 172.21.15.138:0/2811668793 >> v2:172.21.15.190:3300/0 conn(0x7efcd006e0e0 0x7efcd009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 1 -- 172.21.15.138:0/2811668793 >> v1:172.21.15.138:6789/0 conn(0x7efcd009e960 legacy=0x7efcd01356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:35.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient: _finish_hunting 0 2024-09-11T05:26:35.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 1 monclient: found mon.noname-a 2024-09-11T05:26:35.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:35.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.463 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.463+0000 7efcd594e640 1 -- 172.21.15.138:0/2811668793 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd009fb00 con 0x7efcd009ed70 2024-09-11T05:26:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd594e640 1 -- 172.21.15.138:0/2811668793 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd009ed70 msgr2=0x7efcd0135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd594e640 1 --2- 172.21.15.138:0/2811668793 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd009ed70 0x7efcd0135e00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcd0069420 tx=0x7efcc801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fafd5928336a3e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 1 -- 172.21.15.138:0/2811668793 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc80b2070 con 0x7efcd009ed70 2024-09-11T05:26:35.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: dump: 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: _finish_auth 0 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: _check_auth_tickets 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 5 monclient: authenticate success, global_id 114327 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 1 -- 172.21.15.138:0/2811668793 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc80a5940 con 0x7efcd009ed70 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 4 set_mon_vals no callback set 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 1 -- 172.21.15.138:0/2811668793 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc80a5d10 con 0x7efcd009ed70 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcc3fff640 10 monclient: dump: 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.469 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient: shutdown 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- 172.21.15.138:0/2811668793 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd009ed70 msgr2=0x7efcd0135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 --2- 172.21.15.138:0/2811668793 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd009ed70 0x7efcd0135e00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcd0069420 tx=0x7efcc801c880 comp rx=0 tx=0).stop 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- 172.21.15.138:0/2811668793 shutdown_connections 2024-09-11T05:26:35.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 --2- 172.21.15.138:0/2811668793 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd009ed70 0x7efcd0135e00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 --2- 172.21.15.138:0/2811668793 >> v2:172.21.15.190:3300/0 conn(0x7efcd006e0e0 0x7efcd009e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- 172.21.15.138:0/2811668793 >> 172.21.15.138:0/2811668793 conn(0x7efcd0063640 msgr2=0x7efcd0063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- 172.21.15.138:0/2811668793 shutdown_connections 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- 172.21.15.138:0/2811668793 wait complete. 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient: build_initial_monmap 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient: monmap: 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:35.471 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 librados: starting msgr at 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 librados: starting objecter 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 Processor -- start 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- start start 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 librados: setting wanted keys 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 librados: calling monclient init 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient: init 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd006e0e0 0x7efcd0135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient(hunting): picked mon.noname-a con 0x7efcd006e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcd009e960 0x7efcd0135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient(hunting): picked mon.noname-c con 0x7efcd009e960 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd009ed70 0x7efcd011d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient(hunting): picked mon.noname-b con 0x7efcd009ed70 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efcd0136340 con 0x7efcd006e0e0 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efcd01364c0 con 0x7efcd009ed70 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7efcd0140170 con 0x7efcd009e960 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd73b1640 10 monclient(hunting): authenticate will time out at 2078.907442s 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd006e0e0 0x7efcd0135640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd006e0e0 0x7efcd0135640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40856/0 (socket says 172.21.15.138:40856) 2024-09-11T05:26:35.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 -- 172.21.15.138:0/420139312 learned_addr learned my addr 172.21.15.138:0/420139312 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:35.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd494c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcd009e960 0x7efcd0135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient(hunting): get_auth_request con 0x7efcd006e0e0 auth_method 0 2024-09-11T05:26:35.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd494c640 10 monclient(hunting): get_auth_request con 0x7efcd009e960 auth_method 0 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd494c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd494c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd494c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd494c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd494c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd494c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient(hunting): handle_auth_done global_id 114330 payload 824 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd594e640 1 --2- 172.21.15.138:0/420139312 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd009ed70 0x7efcd011d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 -- 172.21.15.138:0/420139312 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcd009e960 msgr2=0x7efcd0135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 --2- 172.21.15.138:0/420139312 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcd009e960 0x7efcd0135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 -- 172.21.15.138:0/420139312 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd009ed70 msgr2=0x7efcd011d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 --2- 172.21.15.138:0/420139312 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd009ed70 0x7efcd011d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient: _finish_hunting 0 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 monclient: found mon.noname-a 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.467+0000 7efcd514d640 1 -- 172.21.15.138:0/420139312 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd0073600 con 0x7efcd006e0e0 2024-09-11T05:26:35.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd514d640 1 -- 172.21.15.138:0/420139312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd006e0e0 msgr2=0x7efcd0135640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd514d640 1 --2- 172.21.15.138:0/420139312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd006e0e0 0x7efcd0135640 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efccc00c7f0 tx=0x7efccc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 client.?.objecter ms_handle_connect 0x7efcd006e0e0 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 1 -- 172.21.15.138:0/420139312 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efccc015070 con 0x7efcd006e0e0 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: dump: 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.479 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: _finish_auth 0 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 1 -- 172.21.15.138:0/420139312 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efccc007a30 con 0x7efcd006e0e0 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 5 monclient: authenticate success, global_id 114330 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 10 monclient: _renew_subs 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd006fa60 con 0x7efcd006e0e0 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 1 -- 172.21.15.138:0/420139312 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efccc007e20 con 0x7efcd006e0e0 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 monclient: dump: 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.481 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 10 client.114330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc37fe640 4 set_mon_vals no callback set 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 10 monclient: _renew_subs 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd01203e0 con 0x7efcd006e0e0 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 1 librados: init done 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd009e420 con 0x7efcd006e0e0 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 1 -- 172.21.15.138:0/420139312 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7efccc0113f0 con 0x7efcd006e0e0 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 1 --2- 172.21.15.138:0/420139312 >> v2:172.21.15.190:6800/1562782189 conn(0x7efcb403ced0 0x7efcb403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 1 -- 172.21.15.138:0/420139312 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7efccc04fa40 con 0x7efcd006e0e0 2024-09-11T05:26:35.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 10 client.114330.objecter ms_dispatch 0x7efcd0000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 3 client.114330.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 3 client.114330.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.471+0000 7efcc1ffb640 20 client.114330.objecter dump_active .. 0 homeless 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.475+0000 7efcd494c640 1 --2- 172.21.15.138:0/420139312 >> v2:172.21.15.190:6800/1562782189 conn(0x7efcb403ced0 0x7efcb403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.475+0000 7efcd494c640 10 monclient: get_auth_request con 0x7efcb403ced0 auth_method 0 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.475+0000 7efcc1ffb640 10 client.114330.objecter ms_handle_connect 0x7efcb403ced0 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.475+0000 7efcd494c640 1 --2- 172.21.15.138:0/420139312 >> v2:172.21.15.190:6800/1562782189 conn(0x7efcb403ced0 0x7efcb403f390 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7efcc4007ca0 tx=0x7efcc4007bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.475+0000 7efcc1ffb640 1 -- 172.21.15.138:0/420139312 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7efccc021cf0 con 0x7efcd006e0e0 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.475+0000 7efcc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.475+0000 7efcc1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:35.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.519+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.519+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571315229}) 2024-09-11T05:26:35.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.519+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1484550857}) 2024-09-11T05:26:35.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.519+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.519+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571315229}) 2024-09-11T05:26:35.515 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.519+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2490049550}) 2024-09-11T05:26:35.641 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.643+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efcd009f270 con 0x7efcb403ced0 2024-09-11T05:26:35.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.651+0000 7efcc1ffb640 1 -- 172.21.15.138:0/420139312 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114112 (secure 0 0 0) 0x7efcc400e6e0 con 0x7efcb403ced0 2024-09-11T05:26:35.649 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:26:35.649 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:26:35.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.651+0000 7efcd73b1640 10 librados: watch_flush enter 2024-09-11T05:26:35.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.651+0000 7efcd73b1640 10 librados: watch_flush exit 2024-09-11T05:26:35.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.651+0000 7efcd73b1640 20 client.114330.objecter shutdown clearing up homeless session... 2024-09-11T05:26:35.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.651+0000 7efcd73b1640 10 client.114330.objecter successfully canceled tick 2024-09-11T05:26:35.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 >> v2:172.21.15.190:6800/1562782189 conn(0x7efcb403ced0 msgr2=0x7efcb403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 --2- 172.21.15.138:0/420139312 >> v2:172.21.15.190:6800/1562782189 conn(0x7efcb403ced0 0x7efcb403f390 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7efcc4007ca0 tx=0x7efcc4007bf0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 10 monclient: shutdown 2024-09-11T05:26:35.651 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd006e0e0 msgr2=0x7efcd0135640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 --2- 172.21.15.138:0/420139312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd006e0e0 0x7efcd0135640 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efccc00c7f0 tx=0x7efccc00ccb0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 shutdown_connections 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 --2- 172.21.15.138:0/420139312 >> v2:172.21.15.190:6800/1562782189 conn(0x7efcb403ced0 0x7efcb403f390 unknown :-1 s=CLOSED pgs=3519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 --2- 172.21.15.138:0/420139312 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd009ed70 0x7efcd011d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 --2- 172.21.15.138:0/420139312 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcd009e960 0x7efcd0135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 --2- 172.21.15.138:0/420139312 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcd006e0e0 0x7efcd0135640 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 >> 172.21.15.138:0/420139312 conn(0x7efcd0063640 msgr2=0x7efcd00a5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 shutdown_connections 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 -- 172.21.15.138:0/420139312 wait complete. 2024-09-11T05:26:35.652 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.655+0000 7efcd73b1640 1 librados: shutdown 2024-09-11T05:26:35.667 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":1260,"stamp":"2024-09-11T05:26:35.061598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":410992,"kb_used_data":21160,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565820048,"statfs":{"total":579820584960,"available":579399729152,"internally_reserved":0,"allocated":21667840,"data_stored":8941820,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":399039137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003678"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.043021+0000","last_change":"2024-09-11T05:08:46.995236+0000","last_active":"2024-09-11T05:24:50.043021+0000","last_peered":"2024-09-11T05:24:50.043021+0000","last_clean":"2024-09-11T05:24:50.043021+0000","last_became_active":"2024-09-11T05:08:46.990401+0000","last_became_peered":"2024-09-11T05:08:46.990401+0000","last_unstale":"2024-09-11T05:24:50.043021+0000","last_undegraded":"2024-09-11T05:24:50.043021+0000","last_fullsized":"2024-09-11T05:24:50.043021+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:17:39.484293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321517+0000","last_change":"2024-09-11T05:08:46.945587+0000","last_active":"2024-09-11T05:24:50.321517+0000","last_peered":"2024-09-11T05:24:50.321517+0000","last_clean":"2024-09-11T05:24:50.321517+0000","last_became_active":"2024-09-11T05:08:46.944961+0000","last_became_peered":"2024-09-11T05:08:46.944961+0000","last_unstale":"2024-09-11T05:24:50.321517+0000","last_undegraded":"2024-09-11T05:24:50.321517+0000","last_fullsized":"2024-09-11T05:24:50.321517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:51:02.865309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328736+0000","last_change":"2024-09-11T05:08:46.948482+0000","last_active":"2024-09-11T05:24:51.328736+0000","last_peered":"2024-09-11T05:24:51.328736+0000","last_clean":"2024-09-11T05:24:51.328736+0000","last_became_active":"2024-09-11T05:08:46.946203+0000","last_became_peered":"2024-09-11T05:08:46.946203+0000","last_unstale":"2024-09-11T05:24:51.328736+0000","last_undegraded":"2024-09-11T05:24:51.328736+0000","last_fullsized":"2024-09-11T05:24:51.328736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:50.073601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427369+0000","last_change":"2024-09-11T05:08:46.872668+0000","last_active":"2024-09-11T05:24:52.427369+0000","last_peered":"2024-09-11T05:24:52.427369+0000","last_clean":"2024-09-11T05:24:52.427369+0000","last_became_active":"2024-09-11T05:08:46.870538+0000","last_became_peered":"2024-09-11T05:08:46.870538+0000","last_unstale":"2024-09-11T05:24:52.427369+0000","last_undegraded":"2024-09-11T05:24:52.427369+0000","last_fullsized":"2024-09-11T05:24:52.427369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:19:55.699319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443776+0000","last_change":"2024-09-11T05:08:46.869521+0000","last_active":"2024-09-11T05:24:51.443776+0000","last_peered":"2024-09-11T05:24:51.443776+0000","last_clean":"2024-09-11T05:24:51.443776+0000","last_became_active":"2024-09-11T05:08:46.866976+0000","last_became_peered":"2024-09-11T05:08:46.866976+0000","last_unstale":"2024-09-11T05:24:51.443776+0000","last_undegraded":"2024-09-11T05:24:51.443776+0000","last_fullsized":"2024-09-11T05:24:51.443776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:52.479759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943518+0000","last_change":"2024-09-11T05:08:46.990679+0000","last_active":"2024-09-11T05:24:52.943518+0000","last_peered":"2024-09-11T05:24:52.943518+0000","last_clean":"2024-09-11T05:24:52.943518+0000","last_became_active":"2024-09-11T05:08:46.989288+0000","last_became_peered":"2024-09-11T05:08:46.989288+0000","last_unstale":"2024-09-11T05:24:52.943518+0000","last_undegraded":"2024-09-11T05:24:52.943518+0000","last_fullsized":"2024-09-11T05:24:52.943518+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:42.357108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426794+0000","last_change":"2024-09-11T05:08:46.896344+0000","last_active":"2024-09-11T05:24:52.426794+0000","last_peered":"2024-09-11T05:24:52.426794+0000","last_clean":"2024-09-11T05:24:52.426794+0000","last_became_active":"2024-09-11T05:08:46.894236+0000","last_became_peered":"2024-09-11T05:08:46.894236+0000","last_unstale":"2024-09-11T05:24:52.426794+0000","last_undegraded":"2024-09-11T05:24:52.426794+0000","last_fullsized":"2024-09-11T05:24:52.426794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:30:58.707091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042731+0000","last_change":"2024-09-11T05:08:46.933931+0000","last_active":"2024-09-11T05:24:50.042731+0000","last_peered":"2024-09-11T05:24:50.042731+0000","last_clean":"2024-09-11T05:24:50.042731+0000","last_became_active":"2024-09-11T05:08:46.925803+0000","last_became_peered":"2024-09-11T05:08:46.925803+0000","last_unstale":"2024-09-11T05:24:50.042731+0000","last_undegraded":"2024-09-11T05:24:50.042731+0000","last_fullsized":"2024-09-11T05:24:50.042731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:47:49.987042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321563+0000","last_change":"2024-09-11T05:08:46.940919+0000","last_active":"2024-09-11T05:24:50.321563+0000","last_peered":"2024-09-11T05:24:50.321563+0000","last_clean":"2024-09-11T05:24:50.321563+0000","last_became_active":"2024-09-11T05:08:46.940655+0000","last_became_peered":"2024-09-11T05:08:46.940655+0000","last_unstale":"2024-09-11T05:24:50.321563+0000","last_undegraded":"2024-09-11T05:24:50.321563+0000","last_fullsized":"2024-09-11T05:24:50.321563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:01:31.299279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328781+0000","last_change":"2024-09-11T05:08:46.995447+0000","last_active":"2024-09-11T05:24:51.328781+0000","last_peered":"2024-09-11T05:24:51.328781+0000","last_clean":"2024-09-11T05:24:51.328781+0000","last_became_active":"2024-09-11T05:08:46.995196+0000","last_became_peered":"2024-09-11T05:08:46.995196+0000","last_unstale":"2024-09-11T05:24:51.328781+0000","last_undegraded":"2024-09-11T05:24:51.328781+0000","last_fullsized":"2024-09-11T05:24:51.328781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:36:36.230686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943683+0000","last_change":"2024-09-11T05:08:46.900776+0000","last_active":"2024-09-11T05:24:52.943683+0000","last_peered":"2024-09-11T05:24:52.943683+0000","last_clean":"2024-09-11T05:24:52.943683+0000","last_became_active":"2024-09-11T05:08:46.898918+0000","last_became_peered":"2024-09-11T05:08:46.898918+0000","last_unstale":"2024-09-11T05:24:52.943683+0000","last_undegraded":"2024-09-11T05:24:52.943683+0000","last_fullsized":"2024-09-11T05:24:52.943683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:07.016324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444013+0000","last_change":"2024-09-11T05:08:46.882790+0000","last_active":"2024-09-11T05:24:51.444013+0000","last_peered":"2024-09-11T05:24:51.444013+0000","last_clean":"2024-09-11T05:24:51.444013+0000","last_became_active":"2024-09-11T05:08:46.882509+0000","last_became_peered":"2024-09-11T05:08:46.882509+0000","last_unstale":"2024-09-11T05:24:51.444013+0000","last_undegraded":"2024-09-11T05:24:51.444013+0000","last_fullsized":"2024-09-11T05:24:51.444013+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:00:58.227654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444282+0000","last_change":"2024-09-11T05:08:46.906621+0000","last_active":"2024-09-11T05:24:51.444282+0000","last_peered":"2024-09-11T05:24:51.444282+0000","last_clean":"2024-09-11T05:24:51.444282+0000","last_became_active":"2024-09-11T05:08:46.897464+0000","last_became_peered":"2024-09-11T05:08:46.897464+0000","last_unstale":"2024-09-11T05:24:51.444282+0000","last_undegraded":"2024-09-11T05:24:51.444282+0000","last_fullsized":"2024-09-11T05:24:51.444282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:01:20.019858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1426,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426997+0000","last_change":"2024-09-11T05:07:43.495116+0000","last_active":"2024-09-11T05:24:52.426997+0000","last_peered":"2024-09-11T05:24:52.426997+0000","last_clean":"2024-09-11T05:24:52.426997+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:24:52.426997+0000","last_undegraded":"2024-09-11T05:24:52.426997+0000","last_fullsized":"2024-09-11T05:24:52.426997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_clean_scrub_stamp":"2024-09-11T05:07:41.703163+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:01.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1335,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443708+0000","last_change":"2024-09-11T05:07:50.540810+0000","last_active":"2024-09-11T05:24:51.443708+0000","last_peered":"2024-09-11T05:24:51.443708+0000","last_clean":"2024-09-11T05:24:51.443708+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:24:51.443708+0000","last_undegraded":"2024-09-11T05:24:51.443708+0000","last_fullsized":"2024-09-11T05:24:51.443708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:39.133232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1323,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944078+0000","last_change":"2024-09-11T05:08:46.872094+0000","last_active":"2024-09-11T05:24:52.944078+0000","last_peered":"2024-09-11T05:24:52.944078+0000","last_clean":"2024-09-11T05:24:52.944078+0000","last_became_active":"2024-09-11T05:08:46.869500+0000","last_became_peered":"2024-09-11T05:08:46.869500+0000","last_unstale":"2024-09-11T05:24:52.944078+0000","last_undegraded":"2024-09-11T05:24:52.944078+0000","last_fullsized":"2024-09-11T05:24:52.944078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:37.860847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1335,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426715+0000","last_change":"2024-09-11T05:07:50.535448+0000","last_active":"2024-09-11T05:24:52.426715+0000","last_peered":"2024-09-11T05:24:52.426715+0000","last_clean":"2024-09-11T05:24:52.426715+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:24:52.426715+0000","last_undegraded":"2024-09-11T05:24:52.426715+0000","last_fullsized":"2024-09-11T05:24:52.426715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:57.570691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444362+0000","last_change":"2024-09-11T05:08:46.858452+0000","last_active":"2024-09-11T05:24:51.444362+0000","last_peered":"2024-09-11T05:24:51.444362+0000","last_clean":"2024-09-11T05:24:51.444362+0000","last_became_active":"2024-09-11T05:08:46.858029+0000","last_became_peered":"2024-09-11T05:08:46.858029+0000","last_unstale":"2024-09-11T05:24:51.444362+0000","last_undegraded":"2024-09-11T05:24:51.444362+0000","last_fullsized":"2024-09-11T05:24:51.444362+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:29:56.500535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1324,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328385+0000","last_change":"2024-09-11T05:07:50.533338+0000","last_active":"2024-09-11T05:24:51.328385+0000","last_peered":"2024-09-11T05:24:51.328385+0000","last_clean":"2024-09-11T05:24:51.328385+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:24:51.328385+0000","last_undegraded":"2024-09-11T05:24:51.328385+0000","last_fullsized":"2024-09-11T05:24:51.328385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:04.573351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1323,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944250+0000","last_change":"2024-09-11T05:08:46.892894+0000","last_active":"2024-09-11T05:24:52.944250+0000","last_peered":"2024-09-11T05:24:52.944250+0000","last_clean":"2024-09-11T05:24:52.944250+0000","last_became_active":"2024-09-11T05:08:46.892242+0000","last_became_peered":"2024-09-11T05:08:46.892242+0000","last_unstale":"2024-09-11T05:24:52.944250+0000","last_undegraded":"2024-09-11T05:24:52.944250+0000","last_fullsized":"2024-09-11T05:24:52.944250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:52.672673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1336,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443623+0000","last_change":"2024-09-11T05:07:50.541650+0000","last_active":"2024-09-11T05:24:51.443623+0000","last_peered":"2024-09-11T05:24:51.443623+0000","last_clean":"2024-09-11T05:24:51.443623+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:24:51.443623+0000","last_undegraded":"2024-09-11T05:24:51.443623+0000","last_fullsized":"2024-09-11T05:24:51.443623+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:47.451082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042761+0000","last_change":"2024-09-11T05:08:46.908002+0000","last_active":"2024-09-11T05:24:50.042761+0000","last_peered":"2024-09-11T05:24:50.042761+0000","last_clean":"2024-09-11T05:24:50.042761+0000","last_became_active":"2024-09-11T05:08:46.907666+0000","last_became_peered":"2024-09-11T05:08:46.907666+0000","last_unstale":"2024-09-11T05:24:50.042761+0000","last_undegraded":"2024-09-11T05:24:50.042761+0000","last_fullsized":"2024-09-11T05:24:50.042761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:33.742345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1338,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943212+0000","last_change":"2024-09-11T05:07:50.543876+0000","last_active":"2024-09-11T05:24:52.943212+0000","last_peered":"2024-09-11T05:24:52.943212+0000","last_clean":"2024-09-11T05:24:52.943212+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:24:52.943212+0000","last_undegraded":"2024-09-11T05:24:52.943212+0000","last_fullsized":"2024-09-11T05:24:52.943212+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:47:12.140333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1336,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943905+0000","last_change":"2024-09-11T05:08:46.989871+0000","last_active":"2024-09-11T05:24:52.943905+0000","last_peered":"2024-09-11T05:24:52.943905+0000","last_clean":"2024-09-11T05:24:52.943905+0000","last_became_active":"2024-09-11T05:08:46.989555+0000","last_became_peered":"2024-09-11T05:08:46.989555+0000","last_unstale":"2024-09-11T05:24:52.943905+0000","last_undegraded":"2024-09-11T05:24:52.943905+0000","last_fullsized":"2024-09-11T05:24:52.943905+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:59:08.330593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1334,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427074+0000","last_change":"2024-09-11T05:07:50.535722+0000","last_active":"2024-09-11T05:24:52.427074+0000","last_peered":"2024-09-11T05:24:52.427074+0000","last_clean":"2024-09-11T05:24:52.427074+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:24:52.427074+0000","last_undegraded":"2024-09-11T05:24:52.427074+0000","last_fullsized":"2024-09-11T05:24:52.427074+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:07.310720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426981+0000","last_change":"2024-09-11T05:08:46.919530+0000","last_active":"2024-09-11T05:24:52.426981+0000","last_peered":"2024-09-11T05:24:52.426981+0000","last_clean":"2024-09-11T05:24:52.426981+0000","last_became_active":"2024-09-11T05:08:46.919277+0000","last_became_peered":"2024-09-11T05:08:46.919277+0000","last_unstale":"2024-09-11T05:24:52.426981+0000","last_undegraded":"2024-09-11T05:24:52.426981+0000","last_fullsized":"2024-09-11T05:24:52.426981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:55:31.708446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1338,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943400+0000","last_change":"2024-09-11T05:07:50.543840+0000","last_active":"2024-09-11T05:24:52.943400+0000","last_peered":"2024-09-11T05:24:52.943400+0000","last_clean":"2024-09-11T05:24:52.943400+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:24:52.943400+0000","last_undegraded":"2024-09-11T05:24:52.943400+0000","last_fullsized":"2024-09-11T05:24:52.943400+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:09.809820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1314,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321491+0000","last_change":"2024-09-11T05:08:46.946319+0000","last_active":"2024-09-11T05:24:50.321491+0000","last_peered":"2024-09-11T05:24:50.321491+0000","last_clean":"2024-09-11T05:24:50.321491+0000","last_became_active":"2024-09-11T05:08:46.946035+0000","last_became_peered":"2024-09-11T05:08:46.946035+0000","last_unstale":"2024-09-11T05:24:50.321491+0000","last_undegraded":"2024-09-11T05:24:50.321491+0000","last_fullsized":"2024-09-11T05:24:50.321491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:58:53.031759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1334,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427259+0000","last_change":"2024-09-11T05:07:50.536048+0000","last_active":"2024-09-11T05:24:52.427259+0000","last_peered":"2024-09-11T05:24:52.427259+0000","last_clean":"2024-09-11T05:24:52.427259+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:24:52.427259+0000","last_undegraded":"2024-09-11T05:24:52.427259+0000","last_fullsized":"2024-09-11T05:24:52.427259+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:33.527347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328466+0000","last_change":"2024-09-11T05:08:46.928091+0000","last_active":"2024-09-11T05:24:51.328466+0000","last_peered":"2024-09-11T05:24:51.328466+0000","last_clean":"2024-09-11T05:24:51.328466+0000","last_became_active":"2024-09-11T05:08:46.927819+0000","last_became_peered":"2024-09-11T05:08:46.927819+0000","last_unstale":"2024-09-11T05:24:51.328466+0000","last_undegraded":"2024-09-11T05:24:51.328466+0000","last_fullsized":"2024-09-11T05:24:51.328466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:52:50.836005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042863+0000","last_change":"2024-09-11T05:08:47.007949+0000","last_active":"2024-09-11T05:24:50.042863+0000","last_peered":"2024-09-11T05:24:50.042863+0000","last_clean":"2024-09-11T05:24:50.042863+0000","last_became_active":"2024-09-11T05:08:47.006300+0000","last_became_peered":"2024-09-11T05:08:47.006300+0000","last_unstale":"2024-09-11T05:24:50.042863+0000","last_undegraded":"2024-09-11T05:24:50.042863+0000","last_fullsized":"2024-09-11T05:24:50.042863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:29:25.170714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328753+0000","last_change":"2024-09-11T05:08:46.998761+0000","last_active":"2024-09-11T05:24:51.328753+0000","last_peered":"2024-09-11T05:24:51.328753+0000","last_clean":"2024-09-11T05:24:51.328753+0000","last_became_active":"2024-09-11T05:08:46.998372+0000","last_became_peered":"2024-09-11T05:08:46.998372+0000","last_unstale":"2024-09-11T05:24:51.328753+0000","last_undegraded":"2024-09-11T05:24:51.328753+0000","last_fullsized":"2024-09-11T05:24:51.328753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:14:12.171086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443982+0000","last_change":"2024-09-11T05:08:46.891774+0000","last_active":"2024-09-11T05:24:51.443982+0000","last_peered":"2024-09-11T05:24:51.443982+0000","last_clean":"2024-09-11T05:24:51.443982+0000","last_became_active":"2024-09-11T05:08:46.890326+0000","last_became_peered":"2024-09-11T05:08:46.890326+0000","last_unstale":"2024-09-11T05:24:51.443982+0000","last_undegraded":"2024-09-11T05:24:51.443982+0000","last_fullsized":"2024-09-11T05:24:51.443982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.028053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944186+0000","last_change":"2024-09-11T05:08:46.872093+0000","last_active":"2024-09-11T05:24:52.944186+0000","last_peered":"2024-09-11T05:24:52.944186+0000","last_clean":"2024-09-11T05:24:52.944186+0000","last_became_active":"2024-09-11T05:08:46.866777+0000","last_became_peered":"2024-09-11T05:08:46.866777+0000","last_unstale":"2024-09-11T05:24:52.944186+0000","last_undegraded":"2024-09-11T05:24:52.944186+0000","last_fullsized":"2024-09-11T05:24:52.944186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:15:57.315991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427125+0000","last_change":"2024-09-11T05:08:46.863954+0000","last_active":"2024-09-11T05:24:52.427125+0000","last_peered":"2024-09-11T05:24:52.427125+0000","last_clean":"2024-09-11T05:24:52.427125+0000","last_became_active":"2024-09-11T05:08:46.858140+0000","last_became_peered":"2024-09-11T05:08:46.858140+0000","last_unstale":"2024-09-11T05:24:52.427125+0000","last_undegraded":"2024-09-11T05:24:52.427125+0000","last_fullsized":"2024-09-11T05:24:52.427125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:30.923117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1309,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328837+0000","last_change":"2024-09-11T05:08:46.945263+0000","last_active":"2024-09-11T05:24:51.328837+0000","last_peered":"2024-09-11T05:24:51.328837+0000","last_clean":"2024-09-11T05:24:51.328837+0000","last_became_active":"2024-09-11T05:08:46.943185+0000","last_became_peered":"2024-09-11T05:08:46.943185+0000","last_unstale":"2024-09-11T05:24:51.328837+0000","last_undegraded":"2024-09-11T05:24:51.328837+0000","last_fullsized":"2024-09-11T05:24:51.328837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:09:14.250455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427004+0000","last_change":"2024-09-11T05:08:46.921575+0000","last_active":"2024-09-11T05:24:52.427004+0000","last_peered":"2024-09-11T05:24:52.427004+0000","last_clean":"2024-09-11T05:24:52.427004+0000","last_became_active":"2024-09-11T05:08:46.916824+0000","last_became_peered":"2024-09-11T05:08:46.916824+0000","last_unstale":"2024-09-11T05:24:52.427004+0000","last_undegraded":"2024-09-11T05:24:52.427004+0000","last_fullsized":"2024-09-11T05:24:52.427004+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:57.444456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443671+0000","last_change":"2024-09-11T05:08:46.843485+0000","last_active":"2024-09-11T05:24:51.443671+0000","last_peered":"2024-09-11T05:24:51.443671+0000","last_clean":"2024-09-11T05:24:51.443671+0000","last_became_active":"2024-09-11T05:08:46.843127+0000","last_became_peered":"2024-09-11T05:08:46.843127+0000","last_unstale":"2024-09-11T05:24:51.443671+0000","last_undegraded":"2024-09-11T05:24:51.443671+0000","last_fullsized":"2024-09-11T05:24:51.443671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:50:14.321453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042475+0000","last_change":"2024-09-11T05:08:46.928312+0000","last_active":"2024-09-11T05:24:50.042475+0000","last_peered":"2024-09-11T05:24:50.042475+0000","last_clean":"2024-09-11T05:24:50.042475+0000","last_became_active":"2024-09-11T05:08:46.925739+0000","last_became_peered":"2024-09-11T05:08:46.925739+0000","last_unstale":"2024-09-11T05:24:50.042475+0000","last_undegraded":"2024-09-11T05:24:50.042475+0000","last_fullsized":"2024-09-11T05:24:50.042475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:12.888188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042613+0000","last_change":"2024-09-11T05:08:46.928128+0000","last_active":"2024-09-11T05:24:50.042613+0000","last_peered":"2024-09-11T05:24:50.042613+0000","last_clean":"2024-09-11T05:24:50.042613+0000","last_became_active":"2024-09-11T05:08:46.925887+0000","last_became_peered":"2024-09-11T05:08:46.925887+0000","last_unstale":"2024-09-11T05:24:50.042613+0000","last_undegraded":"2024-09-11T05:24:50.042613+0000","last_fullsized":"2024-09-11T05:24:50.042613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:27.751280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321512+0000","last_change":"2024-09-11T05:08:46.891878+0000","last_active":"2024-09-11T05:24:50.321512+0000","last_peered":"2024-09-11T05:24:50.321512+0000","last_clean":"2024-09-11T05:24:50.321512+0000","last_became_active":"2024-09-11T05:08:46.889622+0000","last_became_peered":"2024-09-11T05:08:46.889622+0000","last_unstale":"2024-09-11T05:24:50.321512+0000","last_undegraded":"2024-09-11T05:24:50.321512+0000","last_fullsized":"2024-09-11T05:24:50.321512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:22.796907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607782,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66696,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305144,"statfs":{"total":96636764160,"available":96568467456,"internally_reserved":0,"allocated":3653632,"data_stored":1842919,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640485,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74952,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296888,"statfs":{"total":96636764160,"available":96560013312,"internally_reserved":0,"allocated":3592192,"data_stored":1314005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640484,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48584,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323256,"statfs":{"total":96636764160,"available":96587014144,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70792,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301048,"statfs":{"total":96636764160,"available":96564273152,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640485,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66636,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305204,"statfs":{"total":96636764160,"available":96568528896,"internally_reserved":0,"allocated":3596288,"data_stored":1314010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640485,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83332,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94288508,"statfs":{"total":96636764160,"available":96551432192,"internally_reserved":0,"allocated":3649536,"data_stored":1842914,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:26:35.670 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:26:35.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 10 monclient: get_monmap_and_config 2024-09-11T05:26:35.793 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 10 monclient: build_initial_monmap 2024-09-11T05:26:35.794 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 10 monclient: monmap: 2024-09-11T05:26:35.794 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:35.794 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.794 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:35.800005+0000 2024-09-11T05:26:35.794 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:35.800005+0000 2024-09-11T05:26:35.794 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:35.795 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.795 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:35.795 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:35.795 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:35.795 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:35.795 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:35.796 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 1 Processor -- start 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 1 -- start start 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 10 monclient: init 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.795+0000 7f651b7e6640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f651406e0e0 0x7f6514136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 10 monclient(hunting): picked mon.noname-a con 0x7f651406e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6514136170 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6514136c60 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:35.798 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65140a3270 con 0x7f6514136170 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65140a0af0 con 0x7f6514136c60 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f6514137070 con 0x7f651406e0e0 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f651b7e6640 10 monclient(hunting): authenticate will time out at 2079.236105s 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f651406e0e0 0x7f6514136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6518d81640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f6514136170 0x7f65140a1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50024/0 (socket says 172.21.15.138:50024) 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6518d81640 1 -- 172.21.15.138:0/2773556138 learned_addr learned my addr 172.21.15.138:0/2773556138 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient(hunting): get_auth_request con 0x7f651406e0e0 auth_method 0 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.799 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient(hunting): handle_auth_done global_id 114333 payload 274 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 1 -- 172.21.15.138:0/2773556138 >> v1:172.21.15.190:6789/0 conn(0x7f6514136c60 legacy=0x7f65140a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 1 -- 172.21.15.138:0/2773556138 >> v1:172.21.15.138:6789/0 conn(0x7f6514136170 legacy=0x7f65140a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient: _finish_hunting 0 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 1 monclient: found mon.noname-a 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.800 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.799+0000 7f6519582640 1 -- 172.21.15.138:0/2773556138 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65140a2740 con 0x7f651406e0e0 2024-09-11T05:26:35.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519582640 1 -- 172.21.15.138:0/2773556138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f651406e0e0 msgr2=0x7f6514136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:35.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519582640 1 --2- 172.21.15.138:0/2773556138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f651406e0e0 0x7f6514136720 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f65080097c0 tx=0x7f650801c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58859577f250550 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 1 -- 172.21.15.138:0/2773556138 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080b1070 con 0x7f651406e0e0 2024-09-11T05:26:35.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:35.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: dump: 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: _finish_auth 0 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: _check_auth_tickets 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:35.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 5 monclient: authenticate success, global_id 114333 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 1 -- 172.21.15.138:0/2773556138 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65080ac440 con 0x7f651406e0e0 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 1 -- 172.21.15.138:0/2773556138 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080ac810 con 0x7f651406e0e0 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 4 set_mon_vals no callback set 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6507fff640 10 monclient: dump: 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.809 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient: shutdown 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- 172.21.15.138:0/2773556138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f651406e0e0 msgr2=0x7f6514136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 --2- 172.21.15.138:0/2773556138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f651406e0e0 0x7f6514136720 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f65080097c0 tx=0x7f650801c180 comp rx=0 tx=0).stop 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- 172.21.15.138:0/2773556138 shutdown_connections 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 --2- 172.21.15.138:0/2773556138 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f651406e0e0 0x7f6514136720 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- 172.21.15.138:0/2773556138 >> 172.21.15.138:0/2773556138 conn(0x7f6514063640 msgr2=0x7f6514063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:35.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- 172.21.15.138:0/2773556138 shutdown_connections 2024-09-11T05:26:35.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- 172.21.15.138:0/2773556138 wait complete. 2024-09-11T05:26:35.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient: build_initial_monmap 2024-09-11T05:26:35.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient: monmap: 2024-09-11T05:26:35.811 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:35.811 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:35.811 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:35.811 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 librados: starting msgr at 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 librados: starting objecter 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 Processor -- start 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- start start 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 librados: setting wanted keys 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 librados: calling monclient init 2024-09-11T05:26:35.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient: init 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651406e0e0 0x7f65140705d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient(hunting): picked mon.noname-b con 0x7f651406e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6514136170 0x7f6514070b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6514136170 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6514136c60 0x7f6514071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6514136c60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f65141420b0 con 0x7f6514136170 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6514141f30 con 0x7f651406e0e0 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6514142230 con 0x7f6514136c60 2024-09-11T05:26:35.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f651b7e6640 10 monclient(hunting): authenticate will time out at 2079.243091s 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6514136170 0x7f6514070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6514136170 0x7f6514070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40872/0 (socket says 172.21.15.138:40872) 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 -- 172.21.15.138:0/4177762380 learned_addr learned my addr 172.21.15.138:0/4177762380 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient(hunting): get_auth_request con 0x7f6514136170 auth_method 0 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519d83640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6514136c60 0x7f6514071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519d83640 10 monclient(hunting): get_auth_request con 0x7f6514136c60 auth_method 0 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519d83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519d83640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519d83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519d83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519d83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519d83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519582640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651406e0e0 0x7f65140705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519582640 10 monclient(hunting): get_auth_request con 0x7f651406e0e0 auth_method 0 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519582640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:35.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519582640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6519582640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient(hunting): handle_auth_done global_id 114336 payload 824 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 -- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6514136c60 msgr2=0x7f6514071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6514136c60 0x7f6514071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 -- 172.21.15.138:0/4177762380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651406e0e0 msgr2=0x7f65140705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651406e0e0 0x7f65140705d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient: _finish_hunting 0 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 monclient: found mon.noname-a 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.803+0000 7f6518d81640 1 -- 172.21.15.138:0/4177762380 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65140a6c60 con 0x7f6514136170 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6518d81640 1 -- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6514136170 msgr2=0x7f6514070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:35.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6518d81640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6514136170 0x7f6514070b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f650c00cc10 tx=0x7f650c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 client.?.objecter ms_handle_connect 0x7f6514136170 2024-09-11T05:26:35.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:35.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 1 -- 172.21.15.138:0/4177762380 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c012400 con 0x7f6514136170 2024-09-11T05:26:35.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:35.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: dump: 2024-09-11T05:26:35.817 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.817 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: _finish_auth 0 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 1 -- 172.21.15.138:0/4177762380 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f650c0125e0 con 0x7f6514136170 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 5 monclient: authenticate success, global_id 114336 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 10 monclient: _renew_subs 2024-09-11T05:26:35.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6514073cb0 con 0x7f6514136170 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 1 -- 172.21.15.138:0/4177762380 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c0129d0 con 0x7f6514136170 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 monclient: dump: 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:35.819 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6506ffd640 4 set_mon_vals no callback set 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 10 client.114336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 10 monclient: _renew_subs 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6514073730 con 0x7f6514136170 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 1 librados: init done 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6514136720 con 0x7f6514136170 2024-09-11T05:26:35.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 1 -- 172.21.15.138:0/4177762380 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f650c012bf0 con 0x7f6514136170 2024-09-11T05:26:35.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 1 --2- 172.21.15.138:0/4177762380 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64f803cef0 0x7f64f803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:35.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 1 -- 172.21.15.138:0/4177762380 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f650c04fe40 con 0x7f6514136170 2024-09-11T05:26:35.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 10 client.114336.objecter ms_dispatch 0x7f6514000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:35.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 3 client.114336.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:35.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 3 client.114336.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:35.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6505ffb640 20 client.114336.objecter dump_active .. 0 homeless 2024-09-11T05:26:35.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6519582640 1 --2- 172.21.15.138:0/4177762380 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64f803cef0 0x7f64f803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:35.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.807+0000 7f6519582640 10 monclient: get_auth_request con 0x7f64f803cef0 auth_method 0 2024-09-11T05:26:35.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.811+0000 7f6519582640 1 --2- 172.21.15.138:0/4177762380 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64f803cef0 0x7f64f803f3b0 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f6514130830 tx=0x7f6508008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:35.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.811+0000 7f6505ffb640 10 client.114336.objecter ms_handle_connect 0x7f64f803cef0 2024-09-11T05:26:35.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.811+0000 7f6505ffb640 1 -- 172.21.15.138:0/4177762380 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f65140a6c60 con 0x7f6514136170 2024-09-11T05:26:35.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.815+0000 7f6505ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:35.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.815+0000 7f6505ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:35.823 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.823+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.823 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.823+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593556655}) 2024-09-11T05:26:35.823 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.823+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711317223}) 2024-09-11T05:26:35.823 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.823+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.823 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.823+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593556655}) 2024-09-11T05:26:35.823 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.823+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1056877836}) 2024-09-11T05:26:35.825 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.827+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.825 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.827+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775988869}) 2024-09-11T05:26:35.826 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.827+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3541325953}) 2024-09-11T05:26:35.826 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.827+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.826 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.827+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775988869}) 2024-09-11T05:26:35.826 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.827+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=582037071}) 2024-09-11T05:26:35.832 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.835+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.832 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.835+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008016891}) 2024-09-11T05:26:35.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.835+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255208938}) 2024-09-11T05:26:35.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.835+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:35.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.835+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008016891}) 2024-09-11T05:26:35.833 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:35.835+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1488865776}) 2024-09-11T05:26:35.980 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.983+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f651406ccd0 con 0x7f64f803cef0 2024-09-11T05:26:35.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.987+0000 7f6505ffb640 1 -- 172.21.15.138:0/4177762380 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114112 (secure 0 0 0) 0x7f651406ccd0 con 0x7f64f803cef0 2024-09-11T05:26:35.987 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:26:35.988 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:26:35.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 10 librados: watch_flush enter 2024-09-11T05:26:35.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 10 librados: watch_flush exit 2024-09-11T05:26:35.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 20 client.114336.objecter shutdown clearing up homeless session... 2024-09-11T05:26:35.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 10 client.114336.objecter successfully canceled tick 2024-09-11T05:26:35.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64f803cef0 msgr2=0x7f64f803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 --2- 172.21.15.138:0/4177762380 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64f803cef0 0x7f64f803f3b0 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f6514130830 tx=0x7f6508008040 comp rx=0 tx=0).stop 2024-09-11T05:26:35.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 10 monclient: shutdown 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6514136170 msgr2=0x7f6514070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6514136170 0x7f6514070b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f650c00cc10 tx=0x7f650c00a430 comp rx=0 tx=0).stop 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 shutdown_connections 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 --2- 172.21.15.138:0/4177762380 >> v2:172.21.15.190:6800/1562782189 conn(0x7f64f803cef0 0x7f64f803f3b0 unknown :-1 s=CLOSED pgs=3520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6514136c60 0x7f6514071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6514136170 0x7f6514070b10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 --2- 172.21.15.138:0/4177762380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f651406e0e0 0x7f65140705d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 >> 172.21.15.138:0/4177762380 conn(0x7f6514063640 msgr2=0x7f65140a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 shutdown_connections 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 -- 172.21.15.138:0/4177762380 wait complete. 2024-09-11T05:26:35.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:35.991+0000 7f651b7e6640 1 librados: shutdown 2024-09-11T05:26:36.006 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":1260,"stamp":"2024-09-11T05:26:35.061598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":410992,"kb_used_data":21160,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565820048,"statfs":{"total":579820584960,"available":579399729152,"internally_reserved":0,"allocated":21667840,"data_stored":8941820,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":399039137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003678"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.043021+0000","last_change":"2024-09-11T05:08:46.995236+0000","last_active":"2024-09-11T05:24:50.043021+0000","last_peered":"2024-09-11T05:24:50.043021+0000","last_clean":"2024-09-11T05:24:50.043021+0000","last_became_active":"2024-09-11T05:08:46.990401+0000","last_became_peered":"2024-09-11T05:08:46.990401+0000","last_unstale":"2024-09-11T05:24:50.043021+0000","last_undegraded":"2024-09-11T05:24:50.043021+0000","last_fullsized":"2024-09-11T05:24:50.043021+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:17:39.484293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321517+0000","last_change":"2024-09-11T05:08:46.945587+0000","last_active":"2024-09-11T05:24:50.321517+0000","last_peered":"2024-09-11T05:24:50.321517+0000","last_clean":"2024-09-11T05:24:50.321517+0000","last_became_active":"2024-09-11T05:08:46.944961+0000","last_became_peered":"2024-09-11T05:08:46.944961+0000","last_unstale":"2024-09-11T05:24:50.321517+0000","last_undegraded":"2024-09-11T05:24:50.321517+0000","last_fullsized":"2024-09-11T05:24:50.321517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:51:02.865309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328736+0000","last_change":"2024-09-11T05:08:46.948482+0000","last_active":"2024-09-11T05:24:51.328736+0000","last_peered":"2024-09-11T05:24:51.328736+0000","last_clean":"2024-09-11T05:24:51.328736+0000","last_became_active":"2024-09-11T05:08:46.946203+0000","last_became_peered":"2024-09-11T05:08:46.946203+0000","last_unstale":"2024-09-11T05:24:51.328736+0000","last_undegraded":"2024-09-11T05:24:51.328736+0000","last_fullsized":"2024-09-11T05:24:51.328736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:50.073601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427369+0000","last_change":"2024-09-11T05:08:46.872668+0000","last_active":"2024-09-11T05:24:52.427369+0000","last_peered":"2024-09-11T05:24:52.427369+0000","last_clean":"2024-09-11T05:24:52.427369+0000","last_became_active":"2024-09-11T05:08:46.870538+0000","last_became_peered":"2024-09-11T05:08:46.870538+0000","last_unstale":"2024-09-11T05:24:52.427369+0000","last_undegraded":"2024-09-11T05:24:52.427369+0000","last_fullsized":"2024-09-11T05:24:52.427369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:19:55.699319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443776+0000","last_change":"2024-09-11T05:08:46.869521+0000","last_active":"2024-09-11T05:24:51.443776+0000","last_peered":"2024-09-11T05:24:51.443776+0000","last_clean":"2024-09-11T05:24:51.443776+0000","last_became_active":"2024-09-11T05:08:46.866976+0000","last_became_peered":"2024-09-11T05:08:46.866976+0000","last_unstale":"2024-09-11T05:24:51.443776+0000","last_undegraded":"2024-09-11T05:24:51.443776+0000","last_fullsized":"2024-09-11T05:24:51.443776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:52.479759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943518+0000","last_change":"2024-09-11T05:08:46.990679+0000","last_active":"2024-09-11T05:24:52.943518+0000","last_peered":"2024-09-11T05:24:52.943518+0000","last_clean":"2024-09-11T05:24:52.943518+0000","last_became_active":"2024-09-11T05:08:46.989288+0000","last_became_peered":"2024-09-11T05:08:46.989288+0000","last_unstale":"2024-09-11T05:24:52.943518+0000","last_undegraded":"2024-09-11T05:24:52.943518+0000","last_fullsized":"2024-09-11T05:24:52.943518+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:42.357108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426794+0000","last_change":"2024-09-11T05:08:46.896344+0000","last_active":"2024-09-11T05:24:52.426794+0000","last_peered":"2024-09-11T05:24:52.426794+0000","last_clean":"2024-09-11T05:24:52.426794+0000","last_became_active":"2024-09-11T05:08:46.894236+0000","last_became_peered":"2024-09-11T05:08:46.894236+0000","last_unstale":"2024-09-11T05:24:52.426794+0000","last_undegraded":"2024-09-11T05:24:52.426794+0000","last_fullsized":"2024-09-11T05:24:52.426794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:30:58.707091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042731+0000","last_change":"2024-09-11T05:08:46.933931+0000","last_active":"2024-09-11T05:24:50.042731+0000","last_peered":"2024-09-11T05:24:50.042731+0000","last_clean":"2024-09-11T05:24:50.042731+0000","last_became_active":"2024-09-11T05:08:46.925803+0000","last_became_peered":"2024-09-11T05:08:46.925803+0000","last_unstale":"2024-09-11T05:24:50.042731+0000","last_undegraded":"2024-09-11T05:24:50.042731+0000","last_fullsized":"2024-09-11T05:24:50.042731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:47:49.987042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321563+0000","last_change":"2024-09-11T05:08:46.940919+0000","last_active":"2024-09-11T05:24:50.321563+0000","last_peered":"2024-09-11T05:24:50.321563+0000","last_clean":"2024-09-11T05:24:50.321563+0000","last_became_active":"2024-09-11T05:08:46.940655+0000","last_became_peered":"2024-09-11T05:08:46.940655+0000","last_unstale":"2024-09-11T05:24:50.321563+0000","last_undegraded":"2024-09-11T05:24:50.321563+0000","last_fullsized":"2024-09-11T05:24:50.321563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:01:31.299279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328781+0000","last_change":"2024-09-11T05:08:46.995447+0000","last_active":"2024-09-11T05:24:51.328781+0000","last_peered":"2024-09-11T05:24:51.328781+0000","last_clean":"2024-09-11T05:24:51.328781+0000","last_became_active":"2024-09-11T05:08:46.995196+0000","last_became_peered":"2024-09-11T05:08:46.995196+0000","last_unstale":"2024-09-11T05:24:51.328781+0000","last_undegraded":"2024-09-11T05:24:51.328781+0000","last_fullsized":"2024-09-11T05:24:51.328781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:36:36.230686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943683+0000","last_change":"2024-09-11T05:08:46.900776+0000","last_active":"2024-09-11T05:24:52.943683+0000","last_peered":"2024-09-11T05:24:52.943683+0000","last_clean":"2024-09-11T05:24:52.943683+0000","last_became_active":"2024-09-11T05:08:46.898918+0000","last_became_peered":"2024-09-11T05:08:46.898918+0000","last_unstale":"2024-09-11T05:24:52.943683+0000","last_undegraded":"2024-09-11T05:24:52.943683+0000","last_fullsized":"2024-09-11T05:24:52.943683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:07.016324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444013+0000","last_change":"2024-09-11T05:08:46.882790+0000","last_active":"2024-09-11T05:24:51.444013+0000","last_peered":"2024-09-11T05:24:51.444013+0000","last_clean":"2024-09-11T05:24:51.444013+0000","last_became_active":"2024-09-11T05:08:46.882509+0000","last_became_peered":"2024-09-11T05:08:46.882509+0000","last_unstale":"2024-09-11T05:24:51.444013+0000","last_undegraded":"2024-09-11T05:24:51.444013+0000","last_fullsized":"2024-09-11T05:24:51.444013+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:00:58.227654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444282+0000","last_change":"2024-09-11T05:08:46.906621+0000","last_active":"2024-09-11T05:24:51.444282+0000","last_peered":"2024-09-11T05:24:51.444282+0000","last_clean":"2024-09-11T05:24:51.444282+0000","last_became_active":"2024-09-11T05:08:46.897464+0000","last_became_peered":"2024-09-11T05:08:46.897464+0000","last_unstale":"2024-09-11T05:24:51.444282+0000","last_undegraded":"2024-09-11T05:24:51.444282+0000","last_fullsized":"2024-09-11T05:24:51.444282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:01:20.019858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1426,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426997+0000","last_change":"2024-09-11T05:07:43.495116+0000","last_active":"2024-09-11T05:24:52.426997+0000","last_peered":"2024-09-11T05:24:52.426997+0000","last_clean":"2024-09-11T05:24:52.426997+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:24:52.426997+0000","last_undegraded":"2024-09-11T05:24:52.426997+0000","last_fullsized":"2024-09-11T05:24:52.426997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_clean_scrub_stamp":"2024-09-11T05:07:41.703163+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:01.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1335,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443708+0000","last_change":"2024-09-11T05:07:50.540810+0000","last_active":"2024-09-11T05:24:51.443708+0000","last_peered":"2024-09-11T05:24:51.443708+0000","last_clean":"2024-09-11T05:24:51.443708+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:24:51.443708+0000","last_undegraded":"2024-09-11T05:24:51.443708+0000","last_fullsized":"2024-09-11T05:24:51.443708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:39.133232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1323,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944078+0000","last_change":"2024-09-11T05:08:46.872094+0000","last_active":"2024-09-11T05:24:52.944078+0000","last_peered":"2024-09-11T05:24:52.944078+0000","last_clean":"2024-09-11T05:24:52.944078+0000","last_became_active":"2024-09-11T05:08:46.869500+0000","last_became_peered":"2024-09-11T05:08:46.869500+0000","last_unstale":"2024-09-11T05:24:52.944078+0000","last_undegraded":"2024-09-11T05:24:52.944078+0000","last_fullsized":"2024-09-11T05:24:52.944078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:37.860847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1335,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426715+0000","last_change":"2024-09-11T05:07:50.535448+0000","last_active":"2024-09-11T05:24:52.426715+0000","last_peered":"2024-09-11T05:24:52.426715+0000","last_clean":"2024-09-11T05:24:52.426715+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:24:52.426715+0000","last_undegraded":"2024-09-11T05:24:52.426715+0000","last_fullsized":"2024-09-11T05:24:52.426715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:57.570691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444362+0000","last_change":"2024-09-11T05:08:46.858452+0000","last_active":"2024-09-11T05:24:51.444362+0000","last_peered":"2024-09-11T05:24:51.444362+0000","last_clean":"2024-09-11T05:24:51.444362+0000","last_became_active":"2024-09-11T05:08:46.858029+0000","last_became_peered":"2024-09-11T05:08:46.858029+0000","last_unstale":"2024-09-11T05:24:51.444362+0000","last_undegraded":"2024-09-11T05:24:51.444362+0000","last_fullsized":"2024-09-11T05:24:51.444362+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:29:56.500535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1324,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328385+0000","last_change":"2024-09-11T05:07:50.533338+0000","last_active":"2024-09-11T05:24:51.328385+0000","last_peered":"2024-09-11T05:24:51.328385+0000","last_clean":"2024-09-11T05:24:51.328385+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:24:51.328385+0000","last_undegraded":"2024-09-11T05:24:51.328385+0000","last_fullsized":"2024-09-11T05:24:51.328385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:04.573351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1323,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944250+0000","last_change":"2024-09-11T05:08:46.892894+0000","last_active":"2024-09-11T05:24:52.944250+0000","last_peered":"2024-09-11T05:24:52.944250+0000","last_clean":"2024-09-11T05:24:52.944250+0000","last_became_active":"2024-09-11T05:08:46.892242+0000","last_became_peered":"2024-09-11T05:08:46.892242+0000","last_unstale":"2024-09-11T05:24:52.944250+0000","last_undegraded":"2024-09-11T05:24:52.944250+0000","last_fullsized":"2024-09-11T05:24:52.944250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:52.672673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1336,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443623+0000","last_change":"2024-09-11T05:07:50.541650+0000","last_active":"2024-09-11T05:24:51.443623+0000","last_peered":"2024-09-11T05:24:51.443623+0000","last_clean":"2024-09-11T05:24:51.443623+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:24:51.443623+0000","last_undegraded":"2024-09-11T05:24:51.443623+0000","last_fullsized":"2024-09-11T05:24:51.443623+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:47.451082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042761+0000","last_change":"2024-09-11T05:08:46.908002+0000","last_active":"2024-09-11T05:24:50.042761+0000","last_peered":"2024-09-11T05:24:50.042761+0000","last_clean":"2024-09-11T05:24:50.042761+0000","last_became_active":"2024-09-11T05:08:46.907666+0000","last_became_peered":"2024-09-11T05:08:46.907666+0000","last_unstale":"2024-09-11T05:24:50.042761+0000","last_undegraded":"2024-09-11T05:24:50.042761+0000","last_fullsized":"2024-09-11T05:24:50.042761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:33.742345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1338,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943212+0000","last_change":"2024-09-11T05:07:50.543876+0000","last_active":"2024-09-11T05:24:52.943212+0000","last_peered":"2024-09-11T05:24:52.943212+0000","last_clean":"2024-09-11T05:24:52.943212+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:24:52.943212+0000","last_undegraded":"2024-09-11T05:24:52.943212+0000","last_fullsized":"2024-09-11T05:24:52.943212+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:47:12.140333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1336,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943905+0000","last_change":"2024-09-11T05:08:46.989871+0000","last_active":"2024-09-11T05:24:52.943905+0000","last_peered":"2024-09-11T05:24:52.943905+0000","last_clean":"2024-09-11T05:24:52.943905+0000","last_became_active":"2024-09-11T05:08:46.989555+0000","last_became_peered":"2024-09-11T05:08:46.989555+0000","last_unstale":"2024-09-11T05:24:52.943905+0000","last_undegraded":"2024-09-11T05:24:52.943905+0000","last_fullsized":"2024-09-11T05:24:52.943905+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:59:08.330593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1334,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427074+0000","last_change":"2024-09-11T05:07:50.535722+0000","last_active":"2024-09-11T05:24:52.427074+0000","last_peered":"2024-09-11T05:24:52.427074+0000","last_clean":"2024-09-11T05:24:52.427074+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:24:52.427074+0000","last_undegraded":"2024-09-11T05:24:52.427074+0000","last_fullsized":"2024-09-11T05:24:52.427074+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:07.310720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426981+0000","last_change":"2024-09-11T05:08:46.919530+0000","last_active":"2024-09-11T05:24:52.426981+0000","last_peered":"2024-09-11T05:24:52.426981+0000","last_clean":"2024-09-11T05:24:52.426981+0000","last_became_active":"2024-09-11T05:08:46.919277+0000","last_became_peered":"2024-09-11T05:08:46.919277+0000","last_unstale":"2024-09-11T05:24:52.426981+0000","last_undegraded":"2024-09-11T05:24:52.426981+0000","last_fullsized":"2024-09-11T05:24:52.426981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:55:31.708446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1338,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943400+0000","last_change":"2024-09-11T05:07:50.543840+0000","last_active":"2024-09-11T05:24:52.943400+0000","last_peered":"2024-09-11T05:24:52.943400+0000","last_clean":"2024-09-11T05:24:52.943400+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:24:52.943400+0000","last_undegraded":"2024-09-11T05:24:52.943400+0000","last_fullsized":"2024-09-11T05:24:52.943400+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:09.809820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1314,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321491+0000","last_change":"2024-09-11T05:08:46.946319+0000","last_active":"2024-09-11T05:24:50.321491+0000","last_peered":"2024-09-11T05:24:50.321491+0000","last_clean":"2024-09-11T05:24:50.321491+0000","last_became_active":"2024-09-11T05:08:46.946035+0000","last_became_peered":"2024-09-11T05:08:46.946035+0000","last_unstale":"2024-09-11T05:24:50.321491+0000","last_undegraded":"2024-09-11T05:24:50.321491+0000","last_fullsized":"2024-09-11T05:24:50.321491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:58:53.031759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1334,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427259+0000","last_change":"2024-09-11T05:07:50.536048+0000","last_active":"2024-09-11T05:24:52.427259+0000","last_peered":"2024-09-11T05:24:52.427259+0000","last_clean":"2024-09-11T05:24:52.427259+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:24:52.427259+0000","last_undegraded":"2024-09-11T05:24:52.427259+0000","last_fullsized":"2024-09-11T05:24:52.427259+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:33.527347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328466+0000","last_change":"2024-09-11T05:08:46.928091+0000","last_active":"2024-09-11T05:24:51.328466+0000","last_peered":"2024-09-11T05:24:51.328466+0000","last_clean":"2024-09-11T05:24:51.328466+0000","last_became_active":"2024-09-11T05:08:46.927819+0000","last_became_peered":"2024-09-11T05:08:46.927819+0000","last_unstale":"2024-09-11T05:24:51.328466+0000","last_undegraded":"2024-09-11T05:24:51.328466+0000","last_fullsized":"2024-09-11T05:24:51.328466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:52:50.836005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042863+0000","last_change":"2024-09-11T05:08:47.007949+0000","last_active":"2024-09-11T05:24:50.042863+0000","last_peered":"2024-09-11T05:24:50.042863+0000","last_clean":"2024-09-11T05:24:50.042863+0000","last_became_active":"2024-09-11T05:08:47.006300+0000","last_became_peered":"2024-09-11T05:08:47.006300+0000","last_unstale":"2024-09-11T05:24:50.042863+0000","last_undegraded":"2024-09-11T05:24:50.042863+0000","last_fullsized":"2024-09-11T05:24:50.042863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:29:25.170714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328753+0000","last_change":"2024-09-11T05:08:46.998761+0000","last_active":"2024-09-11T05:24:51.328753+0000","last_peered":"2024-09-11T05:24:51.328753+0000","last_clean":"2024-09-11T05:24:51.328753+0000","last_became_active":"2024-09-11T05:08:46.998372+0000","last_became_peered":"2024-09-11T05:08:46.998372+0000","last_unstale":"2024-09-11T05:24:51.328753+0000","last_undegraded":"2024-09-11T05:24:51.328753+0000","last_fullsized":"2024-09-11T05:24:51.328753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:14:12.171086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443982+0000","last_change":"2024-09-11T05:08:46.891774+0000","last_active":"2024-09-11T05:24:51.443982+0000","last_peered":"2024-09-11T05:24:51.443982+0000","last_clean":"2024-09-11T05:24:51.443982+0000","last_became_active":"2024-09-11T05:08:46.890326+0000","last_became_peered":"2024-09-11T05:08:46.890326+0000","last_unstale":"2024-09-11T05:24:51.443982+0000","last_undegraded":"2024-09-11T05:24:51.443982+0000","last_fullsized":"2024-09-11T05:24:51.443982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.028053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944186+0000","last_change":"2024-09-11T05:08:46.872093+0000","last_active":"2024-09-11T05:24:52.944186+0000","last_peered":"2024-09-11T05:24:52.944186+0000","last_clean":"2024-09-11T05:24:52.944186+0000","last_became_active":"2024-09-11T05:08:46.866777+0000","last_became_peered":"2024-09-11T05:08:46.866777+0000","last_unstale":"2024-09-11T05:24:52.944186+0000","last_undegraded":"2024-09-11T05:24:52.944186+0000","last_fullsized":"2024-09-11T05:24:52.944186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:15:57.315991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427125+0000","last_change":"2024-09-11T05:08:46.863954+0000","last_active":"2024-09-11T05:24:52.427125+0000","last_peered":"2024-09-11T05:24:52.427125+0000","last_clean":"2024-09-11T05:24:52.427125+0000","last_became_active":"2024-09-11T05:08:46.858140+0000","last_became_peered":"2024-09-11T05:08:46.858140+0000","last_unstale":"2024-09-11T05:24:52.427125+0000","last_undegraded":"2024-09-11T05:24:52.427125+0000","last_fullsized":"2024-09-11T05:24:52.427125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:30.923117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1309,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328837+0000","last_change":"2024-09-11T05:08:46.945263+0000","last_active":"2024-09-11T05:24:51.328837+0000","last_peered":"2024-09-11T05:24:51.328837+0000","last_clean":"2024-09-11T05:24:51.328837+0000","last_became_active":"2024-09-11T05:08:46.943185+0000","last_became_peered":"2024-09-11T05:08:46.943185+0000","last_unstale":"2024-09-11T05:24:51.328837+0000","last_undegraded":"2024-09-11T05:24:51.328837+0000","last_fullsized":"2024-09-11T05:24:51.328837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:09:14.250455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427004+0000","last_change":"2024-09-11T05:08:46.921575+0000","last_active":"2024-09-11T05:24:52.427004+0000","last_peered":"2024-09-11T05:24:52.427004+0000","last_clean":"2024-09-11T05:24:52.427004+0000","last_became_active":"2024-09-11T05:08:46.916824+0000","last_became_peered":"2024-09-11T05:08:46.916824+0000","last_unstale":"2024-09-11T05:24:52.427004+0000","last_undegraded":"2024-09-11T05:24:52.427004+0000","last_fullsized":"2024-09-11T05:24:52.427004+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:57.444456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443671+0000","last_change":"2024-09-11T05:08:46.843485+0000","last_active":"2024-09-11T05:24:51.443671+0000","last_peered":"2024-09-11T05:24:51.443671+0000","last_clean":"2024-09-11T05:24:51.443671+0000","last_became_active":"2024-09-11T05:08:46.843127+0000","last_became_peered":"2024-09-11T05:08:46.843127+0000","last_unstale":"2024-09-11T05:24:51.443671+0000","last_undegraded":"2024-09-11T05:24:51.443671+0000","last_fullsized":"2024-09-11T05:24:51.443671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:50:14.321453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042475+0000","last_change":"2024-09-11T05:08:46.928312+0000","last_active":"2024-09-11T05:24:50.042475+0000","last_peered":"2024-09-11T05:24:50.042475+0000","last_clean":"2024-09-11T05:24:50.042475+0000","last_became_active":"2024-09-11T05:08:46.925739+0000","last_became_peered":"2024-09-11T05:08:46.925739+0000","last_unstale":"2024-09-11T05:24:50.042475+0000","last_undegraded":"2024-09-11T05:24:50.042475+0000","last_fullsized":"2024-09-11T05:24:50.042475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:12.888188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042613+0000","last_change":"2024-09-11T05:08:46.928128+0000","last_active":"2024-09-11T05:24:50.042613+0000","last_peered":"2024-09-11T05:24:50.042613+0000","last_clean":"2024-09-11T05:24:50.042613+0000","last_became_active":"2024-09-11T05:08:46.925887+0000","last_became_peered":"2024-09-11T05:08:46.925887+0000","last_unstale":"2024-09-11T05:24:50.042613+0000","last_undegraded":"2024-09-11T05:24:50.042613+0000","last_fullsized":"2024-09-11T05:24:50.042613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:27.751280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321512+0000","last_change":"2024-09-11T05:08:46.891878+0000","last_active":"2024-09-11T05:24:50.321512+0000","last_peered":"2024-09-11T05:24:50.321512+0000","last_clean":"2024-09-11T05:24:50.321512+0000","last_became_active":"2024-09-11T05:08:46.889622+0000","last_became_peered":"2024-09-11T05:08:46.889622+0000","last_unstale":"2024-09-11T05:24:50.321512+0000","last_undegraded":"2024-09-11T05:24:50.321512+0000","last_fullsized":"2024-09-11T05:24:50.321512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:22.796907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607782,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66696,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305144,"statfs":{"total":96636764160,"available":96568467456,"internally_reserved":0,"allocated":3653632,"data_stored":1842919,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640485,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74952,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296888,"statfs":{"total":96636764160,"available":96560013312,"internally_reserved":0,"allocated":3592192,"data_stored":1314005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640484,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48584,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323256,"statfs":{"total":96636764160,"available":96587014144,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70792,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301048,"statfs":{"total":96636764160,"available":96564273152,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640485,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66636,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305204,"statfs":{"total":96636764160,"available":96568528896,"internally_reserved":0,"allocated":3596288,"data_stored":1314010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640485,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83332,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94288508,"statfs":{"total":96636764160,"available":96551432192,"internally_reserved":0,"allocated":3649536,"data_stored":1842914,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:26:36.007 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T05:26:36.008 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:26:36.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.123+0000 7f377e0ad640 10 monclient: get_monmap_and_config 2024-09-11T05:26:36.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.123+0000 7f377e0ad640 10 monclient: build_initial_monmap 2024-09-11T05:26:36.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.123+0000 7f377e0ad640 10 monclient: monmap: 2024-09-11T05:26:36.122 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:36.122 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.122 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:36.129004+0000 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:36.129004+0000 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:36.123 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 1 Processor -- start 2024-09-11T05:26:36.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 1 -- start start 2024-09-11T05:26:36.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient: init 2024-09-11T05:26:36.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:36.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-11T05:26:36.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f377806e0e0 0x7f3778136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient(hunting): picked mon.noname-b con 0x7f377806e0e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3778136170 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3778136c60 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37780a3270 con 0x7f3778136c60 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37780a0af0 con 0x7f3778136170 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f377806e4c0 con 0x7f377806e0e0 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f377e0ad640 10 monclient(hunting): authenticate will time out at 2079.565103s 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f3777fff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f3778136c60 0x7f37780a2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:50038/0 (socket says 172.21.15.138:50038) 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f3777fff640 1 -- 172.21.15.138:0/311176954 learned_addr learned my addr 172.21.15.138:0/311176954 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 1 --2- 172.21.15.138:0/311176954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f377806e0e0 0x7f3778136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient(hunting): get_auth_request con 0x7f377806e0e0 auth_method 0 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.126 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient(hunting): handle_auth_done global_id 84635 payload 274 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 1 -- 172.21.15.138:0/311176954 >> v1:172.21.15.190:6789/0 conn(0x7f3778136170 legacy=0x7f37780a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 1 -- 172.21.15.138:0/311176954 >> v1:172.21.15.138:6789/0 conn(0x7f3778136c60 legacy=0x7f37780a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient: _finish_hunting 0 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 1 monclient: found mon.noname-b 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:26:36.127 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.127+0000 7f37777fe640 1 -- 172.21.15.138:0/311176954 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37780a2740 con 0x7f377806e0e0 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f37777fe640 1 -- 172.21.15.138:0/311176954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f377806e0e0 msgr2=0x7f3778136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f37777fe640 1 --2- 172.21.15.138:0/311176954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f377806e0e0 0x7f3778136720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f37680097f0 tx=0x7f376801c1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=744f24759e8a6a03 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 1 -- 172.21.15.138:0/311176954 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37680b0070 con 0x7f377806e0e0 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: dump: 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.130 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: _finish_auth 0 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 5 monclient: authenticate success, global_id 84635 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 1 -- 172.21.15.138:0/311176954 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37680ab600 con 0x7f377806e0e0 2024-09-11T05:26:36.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 1 -- 172.21.15.138:0/311176954 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37680ab9d0 con 0x7f377806e0e0 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f3776ffd640 10 monclient: dump: 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.132 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 4 set_mon_vals no callback set 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 10 monclient: shutdown 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 1 -- 172.21.15.138:0/311176954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f377806e0e0 msgr2=0x7f3778136720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 1 --2- 172.21.15.138:0/311176954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f377806e0e0 0x7f3778136720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f37680097f0 tx=0x7f376801c1c0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 1 -- 172.21.15.138:0/311176954 shutdown_connections 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 1 --2- 172.21.15.138:0/311176954 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f377806e0e0 0x7f3778136720 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.133 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 1 -- 172.21.15.138:0/311176954 >> 172.21.15.138:0/311176954 conn(0x7f3778063640 msgr2=0x7f3778063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:36.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 1 -- 172.21.15.138:0/311176954 shutdown_connections 2024-09-11T05:26:36.134 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.131+0000 7f377e0ad640 1 -- 172.21.15.138:0/311176954 wait complete. 2024-09-11T05:26:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient: build_initial_monmap 2024-09-11T05:26:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient: monmap: 2024-09-11T05:26:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:36.138 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 librados: starting msgr at 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 librados: starting objecter 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 Processor -- start 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 -- start start 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 librados: setting wanted keys 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 librados: calling monclient init 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient: init 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-11T05:26:36.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f377806e0e0 0x7f3778070790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient(hunting): picked mon.noname-a con 0x7f377806e0e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3778136170 0x7f3778070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient(hunting): picked mon.noname-c con 0x7f3778136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3778136c60 0x7f3778071210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient(hunting): picked mon.noname-b con 0x7f3778136c60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f37781420b0 con 0x7f377806e0e0 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3778141f30 con 0x7f3778136c60 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3778142230 con 0x7f3778136170 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:36.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f377e0ad640 10 monclient(hunting): authenticate will time out at 2079.573252s 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f377806e0e0 0x7f3778070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f377806e0e0 0x7f3778070790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40878/0 (socket says 172.21.15.138:40878) 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 1 -- 172.21.15.138:0/17962678 learned_addr learned my addr 172.21.15.138:0/17962678 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 10 monclient(hunting): get_auth_request con 0x7f377806e0e0 auth_method 0 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3778136170 0x7f3778070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient(hunting): get_auth_request con 0x7f3778136170 auth_method 0 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:36.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient(hunting): handle_auth_done global_id 84638 payload 824 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f3777fff640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3778136c60 0x7f3778071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 1 -- 172.21.15.138:0/17962678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3778136c60 msgr2=0x7f3778071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3778136c60 0x7f3778071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 1 -- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f377806e0e0 msgr2=0x7f3778070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f377806e0e0 0x7f3778070790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient: _finish_hunting 0 2024-09-11T05:26:36.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 1 monclient: found mon.noname-c 2024-09-11T05:26:36.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:36.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:36.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f376effd640 1 -- 172.21.15.138:0/17962678 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3778135e00 con 0x7f3778136170 2024-09-11T05:26:36.144 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.135+0000 7f37777fe640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f377806e0e0 0x7f3778070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:36.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f376effd640 1 -- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3778136170 msgr2=0x7f3778070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:36.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f376effd640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3778136170 0x7f3778070cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f376000c3e0 tx=0x7f376000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:36.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 client.?.objecter ms_handle_connect 0x7f3778136170 2024-09-11T05:26:36.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:36.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 1 -- 172.21.15.138:0/17962678 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37600119a0 con 0x7f3778136170 2024-09-11T05:26:36.145 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: dump: 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: _finish_auth 0 2024-09-11T05:26:36.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 1 -- 172.21.15.138:0/17962678 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3760017070 con 0x7f3778136170 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 5 monclient: authenticate success, global_id 84638 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 10 monclient: _renew_subs 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3778073ce0 con 0x7f3778136170 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 1 -- 172.21.15.138:0/17962678 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760011d80 con 0x7f3778136170 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 monclient: dump: 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.147 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 10 client.84638.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 10 monclient: _renew_subs 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3775ffb640 4 set_mon_vals no callback set 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f377809e110 con 0x7f3778136170 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 1 librados: init done 2024-09-11T05:26:36.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3778136720 con 0x7f3778136170 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 1 -- 172.21.15.138:0/17962678 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f376001c460 con 0x7f3778136170 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 1 --2- 172.21.15.138:0/17962678 >> v2:172.21.15.190:6800/1562782189 conn(0x7f375803cef0 0x7f375803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 1 -- 172.21.15.138:0/17962678 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f376004f4b0 con 0x7f3778136170 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 10 client.84638.objecter ms_dispatch 0x7f3778000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 3 client.84638.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 3 client.84638.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f3774ff9640 20 client.84638.objecter dump_active .. 0 homeless 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f37777fe640 1 --2- 172.21.15.138:0/17962678 >> v2:172.21.15.190:6800/1562782189 conn(0x7f375803cef0 0x7f375803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.139+0000 7f37777fe640 10 monclient: get_auth_request con 0x7f375803cef0 auth_method 0 2024-09-11T05:26:36.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.143+0000 7f37777fe640 1 --2- 172.21.15.138:0/17962678 >> v2:172.21.15.190:6800/1562782189 conn(0x7f375803cef0 0x7f375803f3b0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f3768009740 tx=0x7f37680a6000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:36.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.143+0000 7f3774ff9640 10 client.84638.objecter ms_handle_connect 0x7f375803cef0 2024-09-11T05:26:36.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.143+0000 7f3774ff9640 1 -- 172.21.15.138:0/17962678 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3778135e00 con 0x7f3778136170 2024-09-11T05:26:36.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.143+0000 7f3774ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:36.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.143+0000 7f3774ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:36.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117402708}) 2024-09-11T05:26:36.236 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958008616}) 2024-09-11T05:26:36.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117402708}) 2024-09-11T05:26:36.237 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4026204831}) 2024-09-11T05:26:36.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384641085}) 2024-09-11T05:26:36.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3023367411}) 2024-09-11T05:26:36.238 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.239 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384641085}) 2024-09-11T05:26:36.239 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.239+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2674268490}) 2024-09-11T05:26:36.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.315+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3778073ac0 con 0x7f375803cef0 2024-09-11T05:26:36.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.319+0000 7f3774ff9640 1 -- 172.21.15.138:0/17962678 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114112 (secure 0 0 0) 0x7f3778073ac0 con 0x7f375803cef0 2024-09-11T05:26:36.317 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:26:36.318 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 10 librados: watch_flush enter 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 10 librados: watch_flush exit 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 20 client.84638.objecter shutdown clearing up homeless session... 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 10 client.84638.objecter successfully canceled tick 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 >> v2:172.21.15.190:6800/1562782189 conn(0x7f375803cef0 msgr2=0x7f375803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 --2- 172.21.15.138:0/17962678 >> v2:172.21.15.190:6800/1562782189 conn(0x7f375803cef0 0x7f375803f3b0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f3768009740 tx=0x7f37680a6000 comp rx=0 tx=0).stop 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 10 monclient: shutdown 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3778136170 msgr2=0x7f3778070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3778136170 0x7f3778070cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f376000c3e0 tx=0x7f376000c8a0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 shutdown_connections 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 --2- 172.21.15.138:0/17962678 >> v2:172.21.15.190:6800/1562782189 conn(0x7f375803cef0 0x7f375803f3b0 unknown :-1 s=CLOSED pgs=3521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3778136c60 0x7f3778071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3778136170 0x7f3778070cd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 --2- 172.21.15.138:0/17962678 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f377806e0e0 0x7f3778070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 >> 172.21.15.138:0/17962678 conn(0x7f3778063640 msgr2=0x7f37780a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 shutdown_connections 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 -- 172.21.15.138:0/17962678 wait complete. 2024-09-11T05:26:36.322 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.323+0000 7f377e0ad640 1 librados: shutdown 2024-09-11T05:26:36.337 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":1260,"stamp":"2024-09-11T05:26:35.061598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":410992,"kb_used_data":21160,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565820048,"statfs":{"total":579820584960,"available":579399729152,"internally_reserved":0,"allocated":21667840,"data_stored":8941820,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":399039137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003678"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.043021+0000","last_change":"2024-09-11T05:08:46.995236+0000","last_active":"2024-09-11T05:24:50.043021+0000","last_peered":"2024-09-11T05:24:50.043021+0000","last_clean":"2024-09-11T05:24:50.043021+0000","last_became_active":"2024-09-11T05:08:46.990401+0000","last_became_peered":"2024-09-11T05:08:46.990401+0000","last_unstale":"2024-09-11T05:24:50.043021+0000","last_undegraded":"2024-09-11T05:24:50.043021+0000","last_fullsized":"2024-09-11T05:24:50.043021+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:17:39.484293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321517+0000","last_change":"2024-09-11T05:08:46.945587+0000","last_active":"2024-09-11T05:24:50.321517+0000","last_peered":"2024-09-11T05:24:50.321517+0000","last_clean":"2024-09-11T05:24:50.321517+0000","last_became_active":"2024-09-11T05:08:46.944961+0000","last_became_peered":"2024-09-11T05:08:46.944961+0000","last_unstale":"2024-09-11T05:24:50.321517+0000","last_undegraded":"2024-09-11T05:24:50.321517+0000","last_fullsized":"2024-09-11T05:24:50.321517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:51:02.865309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328736+0000","last_change":"2024-09-11T05:08:46.948482+0000","last_active":"2024-09-11T05:24:51.328736+0000","last_peered":"2024-09-11T05:24:51.328736+0000","last_clean":"2024-09-11T05:24:51.328736+0000","last_became_active":"2024-09-11T05:08:46.946203+0000","last_became_peered":"2024-09-11T05:08:46.946203+0000","last_unstale":"2024-09-11T05:24:51.328736+0000","last_undegraded":"2024-09-11T05:24:51.328736+0000","last_fullsized":"2024-09-11T05:24:51.328736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:50.073601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427369+0000","last_change":"2024-09-11T05:08:46.872668+0000","last_active":"2024-09-11T05:24:52.427369+0000","last_peered":"2024-09-11T05:24:52.427369+0000","last_clean":"2024-09-11T05:24:52.427369+0000","last_became_active":"2024-09-11T05:08:46.870538+0000","last_became_peered":"2024-09-11T05:08:46.870538+0000","last_unstale":"2024-09-11T05:24:52.427369+0000","last_undegraded":"2024-09-11T05:24:52.427369+0000","last_fullsized":"2024-09-11T05:24:52.427369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:19:55.699319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443776+0000","last_change":"2024-09-11T05:08:46.869521+0000","last_active":"2024-09-11T05:24:51.443776+0000","last_peered":"2024-09-11T05:24:51.443776+0000","last_clean":"2024-09-11T05:24:51.443776+0000","last_became_active":"2024-09-11T05:08:46.866976+0000","last_became_peered":"2024-09-11T05:08:46.866976+0000","last_unstale":"2024-09-11T05:24:51.443776+0000","last_undegraded":"2024-09-11T05:24:51.443776+0000","last_fullsized":"2024-09-11T05:24:51.443776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:52.479759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943518+0000","last_change":"2024-09-11T05:08:46.990679+0000","last_active":"2024-09-11T05:24:52.943518+0000","last_peered":"2024-09-11T05:24:52.943518+0000","last_clean":"2024-09-11T05:24:52.943518+0000","last_became_active":"2024-09-11T05:08:46.989288+0000","last_became_peered":"2024-09-11T05:08:46.989288+0000","last_unstale":"2024-09-11T05:24:52.943518+0000","last_undegraded":"2024-09-11T05:24:52.943518+0000","last_fullsized":"2024-09-11T05:24:52.943518+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:42.357108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426794+0000","last_change":"2024-09-11T05:08:46.896344+0000","last_active":"2024-09-11T05:24:52.426794+0000","last_peered":"2024-09-11T05:24:52.426794+0000","last_clean":"2024-09-11T05:24:52.426794+0000","last_became_active":"2024-09-11T05:08:46.894236+0000","last_became_peered":"2024-09-11T05:08:46.894236+0000","last_unstale":"2024-09-11T05:24:52.426794+0000","last_undegraded":"2024-09-11T05:24:52.426794+0000","last_fullsized":"2024-09-11T05:24:52.426794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:30:58.707091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042731+0000","last_change":"2024-09-11T05:08:46.933931+0000","last_active":"2024-09-11T05:24:50.042731+0000","last_peered":"2024-09-11T05:24:50.042731+0000","last_clean":"2024-09-11T05:24:50.042731+0000","last_became_active":"2024-09-11T05:08:46.925803+0000","last_became_peered":"2024-09-11T05:08:46.925803+0000","last_unstale":"2024-09-11T05:24:50.042731+0000","last_undegraded":"2024-09-11T05:24:50.042731+0000","last_fullsized":"2024-09-11T05:24:50.042731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:47:49.987042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321563+0000","last_change":"2024-09-11T05:08:46.940919+0000","last_active":"2024-09-11T05:24:50.321563+0000","last_peered":"2024-09-11T05:24:50.321563+0000","last_clean":"2024-09-11T05:24:50.321563+0000","last_became_active":"2024-09-11T05:08:46.940655+0000","last_became_peered":"2024-09-11T05:08:46.940655+0000","last_unstale":"2024-09-11T05:24:50.321563+0000","last_undegraded":"2024-09-11T05:24:50.321563+0000","last_fullsized":"2024-09-11T05:24:50.321563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:01:31.299279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328781+0000","last_change":"2024-09-11T05:08:46.995447+0000","last_active":"2024-09-11T05:24:51.328781+0000","last_peered":"2024-09-11T05:24:51.328781+0000","last_clean":"2024-09-11T05:24:51.328781+0000","last_became_active":"2024-09-11T05:08:46.995196+0000","last_became_peered":"2024-09-11T05:08:46.995196+0000","last_unstale":"2024-09-11T05:24:51.328781+0000","last_undegraded":"2024-09-11T05:24:51.328781+0000","last_fullsized":"2024-09-11T05:24:51.328781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:36:36.230686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943683+0000","last_change":"2024-09-11T05:08:46.900776+0000","last_active":"2024-09-11T05:24:52.943683+0000","last_peered":"2024-09-11T05:24:52.943683+0000","last_clean":"2024-09-11T05:24:52.943683+0000","last_became_active":"2024-09-11T05:08:46.898918+0000","last_became_peered":"2024-09-11T05:08:46.898918+0000","last_unstale":"2024-09-11T05:24:52.943683+0000","last_undegraded":"2024-09-11T05:24:52.943683+0000","last_fullsized":"2024-09-11T05:24:52.943683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:07.016324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444013+0000","last_change":"2024-09-11T05:08:46.882790+0000","last_active":"2024-09-11T05:24:51.444013+0000","last_peered":"2024-09-11T05:24:51.444013+0000","last_clean":"2024-09-11T05:24:51.444013+0000","last_became_active":"2024-09-11T05:08:46.882509+0000","last_became_peered":"2024-09-11T05:08:46.882509+0000","last_unstale":"2024-09-11T05:24:51.444013+0000","last_undegraded":"2024-09-11T05:24:51.444013+0000","last_fullsized":"2024-09-11T05:24:51.444013+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:00:58.227654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444282+0000","last_change":"2024-09-11T05:08:46.906621+0000","last_active":"2024-09-11T05:24:51.444282+0000","last_peered":"2024-09-11T05:24:51.444282+0000","last_clean":"2024-09-11T05:24:51.444282+0000","last_became_active":"2024-09-11T05:08:46.897464+0000","last_became_peered":"2024-09-11T05:08:46.897464+0000","last_unstale":"2024-09-11T05:24:51.444282+0000","last_undegraded":"2024-09-11T05:24:51.444282+0000","last_fullsized":"2024-09-11T05:24:51.444282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:01:20.019858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1426,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426997+0000","last_change":"2024-09-11T05:07:43.495116+0000","last_active":"2024-09-11T05:24:52.426997+0000","last_peered":"2024-09-11T05:24:52.426997+0000","last_clean":"2024-09-11T05:24:52.426997+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:24:52.426997+0000","last_undegraded":"2024-09-11T05:24:52.426997+0000","last_fullsized":"2024-09-11T05:24:52.426997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_clean_scrub_stamp":"2024-09-11T05:07:41.703163+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:01.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1335,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443708+0000","last_change":"2024-09-11T05:07:50.540810+0000","last_active":"2024-09-11T05:24:51.443708+0000","last_peered":"2024-09-11T05:24:51.443708+0000","last_clean":"2024-09-11T05:24:51.443708+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:24:51.443708+0000","last_undegraded":"2024-09-11T05:24:51.443708+0000","last_fullsized":"2024-09-11T05:24:51.443708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:39.133232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1323,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944078+0000","last_change":"2024-09-11T05:08:46.872094+0000","last_active":"2024-09-11T05:24:52.944078+0000","last_peered":"2024-09-11T05:24:52.944078+0000","last_clean":"2024-09-11T05:24:52.944078+0000","last_became_active":"2024-09-11T05:08:46.869500+0000","last_became_peered":"2024-09-11T05:08:46.869500+0000","last_unstale":"2024-09-11T05:24:52.944078+0000","last_undegraded":"2024-09-11T05:24:52.944078+0000","last_fullsized":"2024-09-11T05:24:52.944078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:37.860847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1335,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426715+0000","last_change":"2024-09-11T05:07:50.535448+0000","last_active":"2024-09-11T05:24:52.426715+0000","last_peered":"2024-09-11T05:24:52.426715+0000","last_clean":"2024-09-11T05:24:52.426715+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:24:52.426715+0000","last_undegraded":"2024-09-11T05:24:52.426715+0000","last_fullsized":"2024-09-11T05:24:52.426715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:57.570691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444362+0000","last_change":"2024-09-11T05:08:46.858452+0000","last_active":"2024-09-11T05:24:51.444362+0000","last_peered":"2024-09-11T05:24:51.444362+0000","last_clean":"2024-09-11T05:24:51.444362+0000","last_became_active":"2024-09-11T05:08:46.858029+0000","last_became_peered":"2024-09-11T05:08:46.858029+0000","last_unstale":"2024-09-11T05:24:51.444362+0000","last_undegraded":"2024-09-11T05:24:51.444362+0000","last_fullsized":"2024-09-11T05:24:51.444362+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:29:56.500535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1324,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328385+0000","last_change":"2024-09-11T05:07:50.533338+0000","last_active":"2024-09-11T05:24:51.328385+0000","last_peered":"2024-09-11T05:24:51.328385+0000","last_clean":"2024-09-11T05:24:51.328385+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:24:51.328385+0000","last_undegraded":"2024-09-11T05:24:51.328385+0000","last_fullsized":"2024-09-11T05:24:51.328385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:04.573351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1323,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944250+0000","last_change":"2024-09-11T05:08:46.892894+0000","last_active":"2024-09-11T05:24:52.944250+0000","last_peered":"2024-09-11T05:24:52.944250+0000","last_clean":"2024-09-11T05:24:52.944250+0000","last_became_active":"2024-09-11T05:08:46.892242+0000","last_became_peered":"2024-09-11T05:08:46.892242+0000","last_unstale":"2024-09-11T05:24:52.944250+0000","last_undegraded":"2024-09-11T05:24:52.944250+0000","last_fullsized":"2024-09-11T05:24:52.944250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:52.672673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1336,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443623+0000","last_change":"2024-09-11T05:07:50.541650+0000","last_active":"2024-09-11T05:24:51.443623+0000","last_peered":"2024-09-11T05:24:51.443623+0000","last_clean":"2024-09-11T05:24:51.443623+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:24:51.443623+0000","last_undegraded":"2024-09-11T05:24:51.443623+0000","last_fullsized":"2024-09-11T05:24:51.443623+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:47.451082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042761+0000","last_change":"2024-09-11T05:08:46.908002+0000","last_active":"2024-09-11T05:24:50.042761+0000","last_peered":"2024-09-11T05:24:50.042761+0000","last_clean":"2024-09-11T05:24:50.042761+0000","last_became_active":"2024-09-11T05:08:46.907666+0000","last_became_peered":"2024-09-11T05:08:46.907666+0000","last_unstale":"2024-09-11T05:24:50.042761+0000","last_undegraded":"2024-09-11T05:24:50.042761+0000","last_fullsized":"2024-09-11T05:24:50.042761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:33.742345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1338,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943212+0000","last_change":"2024-09-11T05:07:50.543876+0000","last_active":"2024-09-11T05:24:52.943212+0000","last_peered":"2024-09-11T05:24:52.943212+0000","last_clean":"2024-09-11T05:24:52.943212+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:24:52.943212+0000","last_undegraded":"2024-09-11T05:24:52.943212+0000","last_fullsized":"2024-09-11T05:24:52.943212+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:47:12.140333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1336,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943905+0000","last_change":"2024-09-11T05:08:46.989871+0000","last_active":"2024-09-11T05:24:52.943905+0000","last_peered":"2024-09-11T05:24:52.943905+0000","last_clean":"2024-09-11T05:24:52.943905+0000","last_became_active":"2024-09-11T05:08:46.989555+0000","last_became_peered":"2024-09-11T05:08:46.989555+0000","last_unstale":"2024-09-11T05:24:52.943905+0000","last_undegraded":"2024-09-11T05:24:52.943905+0000","last_fullsized":"2024-09-11T05:24:52.943905+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:59:08.330593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1334,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427074+0000","last_change":"2024-09-11T05:07:50.535722+0000","last_active":"2024-09-11T05:24:52.427074+0000","last_peered":"2024-09-11T05:24:52.427074+0000","last_clean":"2024-09-11T05:24:52.427074+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:24:52.427074+0000","last_undegraded":"2024-09-11T05:24:52.427074+0000","last_fullsized":"2024-09-11T05:24:52.427074+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:07.310720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426981+0000","last_change":"2024-09-11T05:08:46.919530+0000","last_active":"2024-09-11T05:24:52.426981+0000","last_peered":"2024-09-11T05:24:52.426981+0000","last_clean":"2024-09-11T05:24:52.426981+0000","last_became_active":"2024-09-11T05:08:46.919277+0000","last_became_peered":"2024-09-11T05:08:46.919277+0000","last_unstale":"2024-09-11T05:24:52.426981+0000","last_undegraded":"2024-09-11T05:24:52.426981+0000","last_fullsized":"2024-09-11T05:24:52.426981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:55:31.708446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1338,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943400+0000","last_change":"2024-09-11T05:07:50.543840+0000","last_active":"2024-09-11T05:24:52.943400+0000","last_peered":"2024-09-11T05:24:52.943400+0000","last_clean":"2024-09-11T05:24:52.943400+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:24:52.943400+0000","last_undegraded":"2024-09-11T05:24:52.943400+0000","last_fullsized":"2024-09-11T05:24:52.943400+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:09.809820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1314,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321491+0000","last_change":"2024-09-11T05:08:46.946319+0000","last_active":"2024-09-11T05:24:50.321491+0000","last_peered":"2024-09-11T05:24:50.321491+0000","last_clean":"2024-09-11T05:24:50.321491+0000","last_became_active":"2024-09-11T05:08:46.946035+0000","last_became_peered":"2024-09-11T05:08:46.946035+0000","last_unstale":"2024-09-11T05:24:50.321491+0000","last_undegraded":"2024-09-11T05:24:50.321491+0000","last_fullsized":"2024-09-11T05:24:50.321491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:58:53.031759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1334,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427259+0000","last_change":"2024-09-11T05:07:50.536048+0000","last_active":"2024-09-11T05:24:52.427259+0000","last_peered":"2024-09-11T05:24:52.427259+0000","last_clean":"2024-09-11T05:24:52.427259+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:24:52.427259+0000","last_undegraded":"2024-09-11T05:24:52.427259+0000","last_fullsized":"2024-09-11T05:24:52.427259+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:33.527347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328466+0000","last_change":"2024-09-11T05:08:46.928091+0000","last_active":"2024-09-11T05:24:51.328466+0000","last_peered":"2024-09-11T05:24:51.328466+0000","last_clean":"2024-09-11T05:24:51.328466+0000","last_became_active":"2024-09-11T05:08:46.927819+0000","last_became_peered":"2024-09-11T05:08:46.927819+0000","last_unstale":"2024-09-11T05:24:51.328466+0000","last_undegraded":"2024-09-11T05:24:51.328466+0000","last_fullsized":"2024-09-11T05:24:51.328466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:52:50.836005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042863+0000","last_change":"2024-09-11T05:08:47.007949+0000","last_active":"2024-09-11T05:24:50.042863+0000","last_peered":"2024-09-11T05:24:50.042863+0000","last_clean":"2024-09-11T05:24:50.042863+0000","last_became_active":"2024-09-11T05:08:47.006300+0000","last_became_peered":"2024-09-11T05:08:47.006300+0000","last_unstale":"2024-09-11T05:24:50.042863+0000","last_undegraded":"2024-09-11T05:24:50.042863+0000","last_fullsized":"2024-09-11T05:24:50.042863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:29:25.170714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328753+0000","last_change":"2024-09-11T05:08:46.998761+0000","last_active":"2024-09-11T05:24:51.328753+0000","last_peered":"2024-09-11T05:24:51.328753+0000","last_clean":"2024-09-11T05:24:51.328753+0000","last_became_active":"2024-09-11T05:08:46.998372+0000","last_became_peered":"2024-09-11T05:08:46.998372+0000","last_unstale":"2024-09-11T05:24:51.328753+0000","last_undegraded":"2024-09-11T05:24:51.328753+0000","last_fullsized":"2024-09-11T05:24:51.328753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:14:12.171086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443982+0000","last_change":"2024-09-11T05:08:46.891774+0000","last_active":"2024-09-11T05:24:51.443982+0000","last_peered":"2024-09-11T05:24:51.443982+0000","last_clean":"2024-09-11T05:24:51.443982+0000","last_became_active":"2024-09-11T05:08:46.890326+0000","last_became_peered":"2024-09-11T05:08:46.890326+0000","last_unstale":"2024-09-11T05:24:51.443982+0000","last_undegraded":"2024-09-11T05:24:51.443982+0000","last_fullsized":"2024-09-11T05:24:51.443982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.028053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944186+0000","last_change":"2024-09-11T05:08:46.872093+0000","last_active":"2024-09-11T05:24:52.944186+0000","last_peered":"2024-09-11T05:24:52.944186+0000","last_clean":"2024-09-11T05:24:52.944186+0000","last_became_active":"2024-09-11T05:08:46.866777+0000","last_became_peered":"2024-09-11T05:08:46.866777+0000","last_unstale":"2024-09-11T05:24:52.944186+0000","last_undegraded":"2024-09-11T05:24:52.944186+0000","last_fullsized":"2024-09-11T05:24:52.944186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:15:57.315991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427125+0000","last_change":"2024-09-11T05:08:46.863954+0000","last_active":"2024-09-11T05:24:52.427125+0000","last_peered":"2024-09-11T05:24:52.427125+0000","last_clean":"2024-09-11T05:24:52.427125+0000","last_became_active":"2024-09-11T05:08:46.858140+0000","last_became_peered":"2024-09-11T05:08:46.858140+0000","last_unstale":"2024-09-11T05:24:52.427125+0000","last_undegraded":"2024-09-11T05:24:52.427125+0000","last_fullsized":"2024-09-11T05:24:52.427125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:30.923117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1309,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328837+0000","last_change":"2024-09-11T05:08:46.945263+0000","last_active":"2024-09-11T05:24:51.328837+0000","last_peered":"2024-09-11T05:24:51.328837+0000","last_clean":"2024-09-11T05:24:51.328837+0000","last_became_active":"2024-09-11T05:08:46.943185+0000","last_became_peered":"2024-09-11T05:08:46.943185+0000","last_unstale":"2024-09-11T05:24:51.328837+0000","last_undegraded":"2024-09-11T05:24:51.328837+0000","last_fullsized":"2024-09-11T05:24:51.328837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:09:14.250455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427004+0000","last_change":"2024-09-11T05:08:46.921575+0000","last_active":"2024-09-11T05:24:52.427004+0000","last_peered":"2024-09-11T05:24:52.427004+0000","last_clean":"2024-09-11T05:24:52.427004+0000","last_became_active":"2024-09-11T05:08:46.916824+0000","last_became_peered":"2024-09-11T05:08:46.916824+0000","last_unstale":"2024-09-11T05:24:52.427004+0000","last_undegraded":"2024-09-11T05:24:52.427004+0000","last_fullsized":"2024-09-11T05:24:52.427004+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:57.444456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443671+0000","last_change":"2024-09-11T05:08:46.843485+0000","last_active":"2024-09-11T05:24:51.443671+0000","last_peered":"2024-09-11T05:24:51.443671+0000","last_clean":"2024-09-11T05:24:51.443671+0000","last_became_active":"2024-09-11T05:08:46.843127+0000","last_became_peered":"2024-09-11T05:08:46.843127+0000","last_unstale":"2024-09-11T05:24:51.443671+0000","last_undegraded":"2024-09-11T05:24:51.443671+0000","last_fullsized":"2024-09-11T05:24:51.443671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:50:14.321453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042475+0000","last_change":"2024-09-11T05:08:46.928312+0000","last_active":"2024-09-11T05:24:50.042475+0000","last_peered":"2024-09-11T05:24:50.042475+0000","last_clean":"2024-09-11T05:24:50.042475+0000","last_became_active":"2024-09-11T05:08:46.925739+0000","last_became_peered":"2024-09-11T05:08:46.925739+0000","last_unstale":"2024-09-11T05:24:50.042475+0000","last_undegraded":"2024-09-11T05:24:50.042475+0000","last_fullsized":"2024-09-11T05:24:50.042475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:12.888188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042613+0000","last_change":"2024-09-11T05:08:46.928128+0000","last_active":"2024-09-11T05:24:50.042613+0000","last_peered":"2024-09-11T05:24:50.042613+0000","last_clean":"2024-09-11T05:24:50.042613+0000","last_became_active":"2024-09-11T05:08:46.925887+0000","last_became_peered":"2024-09-11T05:08:46.925887+0000","last_unstale":"2024-09-11T05:24:50.042613+0000","last_undegraded":"2024-09-11T05:24:50.042613+0000","last_fullsized":"2024-09-11T05:24:50.042613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:27.751280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321512+0000","last_change":"2024-09-11T05:08:46.891878+0000","last_active":"2024-09-11T05:24:50.321512+0000","last_peered":"2024-09-11T05:24:50.321512+0000","last_clean":"2024-09-11T05:24:50.321512+0000","last_became_active":"2024-09-11T05:08:46.889622+0000","last_became_peered":"2024-09-11T05:08:46.889622+0000","last_unstale":"2024-09-11T05:24:50.321512+0000","last_undegraded":"2024-09-11T05:24:50.321512+0000","last_fullsized":"2024-09-11T05:24:50.321512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:22.796907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607782,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66696,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305144,"statfs":{"total":96636764160,"available":96568467456,"internally_reserved":0,"allocated":3653632,"data_stored":1842919,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640485,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74952,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296888,"statfs":{"total":96636764160,"available":96560013312,"internally_reserved":0,"allocated":3592192,"data_stored":1314005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640484,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48584,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323256,"statfs":{"total":96636764160,"available":96587014144,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640488,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70792,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301048,"statfs":{"total":96636764160,"available":96564273152,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640485,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66636,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305204,"statfs":{"total":96636764160,"available":96568528896,"internally_reserved":0,"allocated":3596288,"data_stored":1314010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640485,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83332,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94288508,"statfs":{"total":96636764160,"available":96551432192,"internally_reserved":0,"allocated":3649536,"data_stored":1842914,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:26:36.339 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T05:26:36.460 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient: get_monmap_and_config 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient: build_initial_monmap 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient: monmap: 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:36.467385+0000 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:36.467385+0000 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:36.461 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:36.462 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:36.462 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 1 Processor -- start 2024-09-11T05:26:36.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 1 -- start start 2024-09-11T05:26:36.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient: init 2024-09-11T05:26:36.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:36.464 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-11T05:26:36.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7fdf4806e0e0 0x7fdf48136720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient(hunting): picked mon.noname-a con 0x7fdf4806e0e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:36.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdf48136170 0x7fdf48136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient(hunting): picked mon.noname-c con 0x7fdf48136170 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:36.465 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf480a1920 0x7fdf480a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient(hunting): picked mon.noname-b con 0x7fdf480a1920 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:36.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7fdf4806e4c0 con 0x7fdf4806e0e0 2024-09-11T05:26:36.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdf480a3270 con 0x7fdf48136170 2024-09-11T05:26:36.466 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdf480a0af0 con 0x7fdf480a1920 2024-09-11T05:26:36.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:36.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4ce43640 10 monclient(hunting): authenticate will time out at 2079.903486s 2024-09-11T05:26:36.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf480a1920 0x7fdf480a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf480a1920 0x7fdf480a1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55636/0 (socket says 172.21.15.138:55636) 2024-09-11T05:26:36.467 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 1 -- 172.21.15.138:0/1226506656 learned_addr learned my addr 172.21.15.138:0/1226506656 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4659c640 1 --2- 172.21.15.138:0/1226506656 >> v2:172.21.15.138:3300/0 conn(0x7fdf4806e0e0 0x7fdf48136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 10 monclient(hunting): get_auth_request con 0x7fdf480a1920 auth_method 0 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4659c640 10 monclient(hunting): get_auth_request con 0x7fdf4806e0e0 auth_method 0 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:36.468 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:36.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf45d9b640 1 --2- 172.21.15.138:0/1226506656 >> v2:172.21.15.190:3300/0 conn(0x7fdf48136170 0x7fdf48136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:36.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.463+0000 7fdf4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:36.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:36.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf45d9b640 10 monclient(hunting): get_auth_request con 0x7fdf48136170 auth_method 0 2024-09-11T05:26:36.469 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 10 monclient(hunting): handle_auth_done global_id 84641 payload 274 2024-09-11T05:26:36.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 1 -- 172.21.15.138:0/1226506656 >> v2:172.21.15.190:3300/0 conn(0x7fdf48136170 msgr2=0x7fdf48136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:36.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 1 --2- 172.21.15.138:0/1226506656 >> v2:172.21.15.190:3300/0 conn(0x7fdf48136170 0x7fdf48136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 1 -- 172.21.15.138:0/1226506656 >> v2:172.21.15.138:3300/0 conn(0x7fdf4806e0e0 msgr2=0x7fdf48136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:36.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 1 --2- 172.21.15.138:0/1226506656 >> v2:172.21.15.138:3300/0 conn(0x7fdf4806e0e0 0x7fdf48136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:26:36.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 1 monclient: found mon.noname-b 2024-09-11T05:26:36.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:36.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:26:36.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf46d9d640 1 -- 172.21.15.138:0/1226506656 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf480a2f90 con 0x7fdf480a1920 2024-09-11T05:26:36.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.467+0000 7fdf4659c640 1 --2- 172.21.15.138:0/1226506656 >> v2:172.21.15.138:3300/0 conn(0x7fdf4806e0e0 0x7fdf48136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:36.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 1 -- 172.21.15.138:0/1226506656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf480a1920 msgr2=0x7fdf480a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:36.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 1 --2- 172.21.15.138:0/1226506656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf480a1920 0x7fdf480a1d70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdf48130830 tx=0x7fdf4001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aea584112ea682 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:36.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 1 -- 172.21.15.138:0/1226506656 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf400090d0 con 0x7fdf480a1920 2024-09-11T05:26:36.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:36.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:26:36.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: dump: 2024-09-11T05:26:36.482 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:36.482 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.483 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.483 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.483 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:36.483 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.483 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:36.483 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:36.484 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:36.484 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: _finish_auth 0 2024-09-11T05:26:36.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: _check_auth_tickets 2024-09-11T05:26:36.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:36.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 1 -- 172.21.15.138:0/1226506656 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf400a64e0 con 0x7fdf480a1920 2024-09-11T05:26:36.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 5 monclient: authenticate success, global_id 84641 2024-09-11T05:26:36.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:36.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:36.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 1 -- 172.21.15.138:0/1226506656 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf40009270 con 0x7fdf480a1920 2024-09-11T05:26:36.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:36.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 4 set_mon_vals no callback set 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4559a640 10 monclient: dump: 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:36.486 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:36.487 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient: shutdown 2024-09-11T05:26:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- 172.21.15.138:0/1226506656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf480a1920 msgr2=0x7fdf480a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/1226506656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf480a1920 0x7fdf480a1d70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdf48130830 tx=0x7fdf4001c8b0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- 172.21.15.138:0/1226506656 shutdown_connections 2024-09-11T05:26:36.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/1226506656 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf480a1920 0x7fdf480a1d70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/1226506656 >> v2:172.21.15.190:3300/0 conn(0x7fdf48136170 0x7fdf48136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/1226506656 >> v2:172.21.15.138:3300/0 conn(0x7fdf4806e0e0 0x7fdf48136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- 172.21.15.138:0/1226506656 >> 172.21.15.138:0/1226506656 conn(0x7fdf48063640 msgr2=0x7fdf48063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:36.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- 172.21.15.138:0/1226506656 shutdown_connections 2024-09-11T05:26:36.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- 172.21.15.138:0/1226506656 wait complete. 2024-09-11T05:26:36.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient: build_initial_monmap 2024-09-11T05:26:36.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient: monmap: 2024-09-11T05:26:36.489 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:36.489 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:36.489 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:36.489 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:36.490 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:36.490 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.490 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:36.490 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:36.490 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:36.490 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 librados: starting msgr at 2024-09-11T05:26:36.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 librados: starting objecter 2024-09-11T05:26:36.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 Processor -- start 2024-09-11T05:26:36.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- start start 2024-09-11T05:26:36.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 librados: setting wanted keys 2024-09-11T05:26:36.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 librados: calling monclient init 2024-09-11T05:26:36.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient: init 2024-09-11T05:26:36.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:36.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:26:36.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf4806e0e0 0x7fdf4813c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient(hunting): picked mon.noname-b con 0x7fdf4806e0e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:36.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf480a1920 0x7fdf4813cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient(hunting): picked mon.noname-a con 0x7fdf480a1920 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:36.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf48136170 0x7fdf48137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient(hunting): picked mon.noname-c con 0x7fdf48136170 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:36.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48142950 con 0x7fdf480a1920 2024-09-11T05:26:36.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481427d0 con 0x7fdf4806e0e0 2024-09-11T05:26:36.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:36.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdf48142ad0 con 0x7fdf48136170 2024-09-11T05:26:36.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:36.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4ce43640 10 monclient(hunting): authenticate will time out at 2079.910806s 2024-09-11T05:26:36.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf480a1920 0x7fdf4813cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf48136170 0x7fdf48137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf480a1920 0x7fdf4813cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:40900/0 (socket says 172.21.15.138:40900) 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 -- 172.21.15.138:0/3060549805 learned_addr learned my addr 172.21.15.138:0/3060549805 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient(hunting): get_auth_request con 0x7fdf480a1920 auth_method 0 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 10 monclient(hunting): get_auth_request con 0x7fdf48136170 auth_method 0 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:36.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4659c640 1 --2- 172.21.15.138:0/3060549805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf4806e0e0 0x7fdf4813c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4659c640 10 monclient(hunting): get_auth_request con 0x7fdf4806e0e0 auth_method 0 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient(hunting): handle_auth_done global_id 114345 payload 824 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 -- 172.21.15.138:0/3060549805 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf48136170 msgr2=0x7fdf48137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 --2- 172.21.15.138:0/3060549805 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf48136170 0x7fdf48137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 -- 172.21.15.138:0/3060549805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf4806e0e0 msgr2=0x7fdf4813c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 --2- 172.21.15.138:0/3060549805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf4806e0e0 0x7fdf4813c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient: _finish_hunting 0 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 monclient: found mon.noname-a 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.471+0000 7fdf45d9b640 1 -- 172.21.15.138:0/3060549805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4813b110 con 0x7fdf480a1920 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf45d9b640 1 -- 172.21.15.138:0/3060549805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf480a1920 msgr2=0x7fdf4813cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf45d9b640 1 --2- 172.21.15.138:0/3060549805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf480a1920 0x7fdf4813cb80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00d800 tx=0x7fdf3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 client.?.objecter ms_handle_connect 0x7fdf480a1920 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 1 -- 172.21.15.138:0/3060549805 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c00b850 con 0x7fdf480a1920 2024-09-11T05:26:36.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: dump: 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: _finish_auth 0 2024-09-11T05:26:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 1 -- 172.21.15.138:0/3060549805 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf3c019070 con 0x7fdf480a1920 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 5 monclient: authenticate success, global_id 114345 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 10 monclient: _renew_subs 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4809e110 con 0x7fdf480a1920 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 1 -- 172.21.15.138:0/3060549805 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c00bc70 con 0x7fdf480a1920 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 monclient: dump: 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 10 client.114345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 10 monclient: _renew_subs 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf48138db0 con 0x7fdf480a1920 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 1 librados: init done 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf44d99640 4 set_mon_vals no callback set 2024-09-11T05:26:36.500 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf480a1d70 con 0x7fdf480a1920 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 1 -- 172.21.15.138:0/3060549805 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdf3c013070 con 0x7fdf480a1920 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 1 --2- 172.21.15.138:0/3060549805 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdf2803cef0 0x7fdf2803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 1 -- 172.21.15.138:0/3060549805 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7fdf3c04f770 con 0x7fdf480a1920 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 10 client.114345.objecter ms_dispatch 0x7fdf48000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 3 client.114345.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 3 client.114345.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf36ffd640 20 client.114345.objecter dump_active .. 0 homeless 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4659c640 1 --2- 172.21.15.138:0/3060549805 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdf2803cef0 0x7fdf2803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.475+0000 7fdf4659c640 10 monclient: get_auth_request con 0x7fdf2803cef0 auth_method 0 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.479+0000 7fdf36ffd640 10 client.114345.objecter ms_handle_connect 0x7fdf2803cef0 2024-09-11T05:26:36.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.479+0000 7fdf4659c640 1 --2- 172.21.15.138:0/3060549805 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdf2803cef0 0x7fdf2803f3b0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009a50 tx=0x7fdf38008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:36.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.479+0000 7fdf36ffd640 1 -- 172.21.15.138:0/3060549805 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fdf3c00be10 con 0x7fdf480a1920 2024-09-11T05:26:36.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.479+0000 7fdf36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:36.502 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.479+0000 7fdf36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:36.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.651+0000 7fdf4ce43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:26:36.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.651+0000 7fdf4ce43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:26:36.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.651+0000 7fdf4ce43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:36.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.651+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdf48139910 con 0x7fdf480a1920 2024-09-11T05:26:36.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.651+0000 7fdf36ffd640 1 -- 172.21.15.138:0/3060549805 <== mon.0 v2:172.21.15.138:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v698) ==== 74+0+14070 (secure 0 0 0) 0x7fdf3c04f030 con 0x7fdf480a1920 2024-09-11T05:26:36.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.651+0000 7fdf36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T05:26:36.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.651+0000 7fdf36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:26:36.651 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:26:36.651 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":698,"fsid":"66c6f018-f82d-4af3-8f0d-6530cdae2ec3","created":"2024-09-11T05:07:27.386635+0000","modified":"2024-09-11T05:24:50.030829+0000","last_up_change":"2024-09-11T05:07:40.695452+0000","last_in_change":"2024-09-11T05:07:31.585387+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":109,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T05:07:40.718263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T05:07:46.826732+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-11T05:08:45.675472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79670fd2-b5aa-408e-8aa4-18a8fb848972","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":693,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":16171034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":16171034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":16171034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":16171034}]},"public_addr":"172.21.15.138:6800/16171034","cluster_addr":"172.21.15.138:6801/16171034","heartbeat_back_addr":"172.21.15.138:6803/16171034","heartbeat_front_addr":"172.21.15.138:6802/16171034","state":["exists","up"]},{"osd":1,"uuid":"e96fe3a6-3058-4c40-aa87-26a89572cdfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":693,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":4002974512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4002974512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":4002974512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":4002974512}]},"public_addr":"172.21.15.138:6808/4002974512","cluster_addr":"172.21.15.138:6809/4002974512","heartbeat_back_addr":"172.21.15.138:6811/4002974512","heartbeat_front_addr":"172.21.15.138:6810/4002974512","state":["exists","up"]},{"osd":2,"uuid":"a87dbce5-b78c-4070-90cc-d323fc444f4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":693,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":81047329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":81047329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":81047329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":81047329}]},"public_addr":"172.21.15.138:6804/81047329","cluster_addr":"172.21.15.138:6805/81047329","heartbeat_back_addr":"172.21.15.138:6807/81047329","heartbeat_front_addr":"172.21.15.138:6806/81047329","state":["exists","up"]},{"osd":3,"uuid":"e1a7ea3f-62f5-4d1f-b671-be183143eb20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":693,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":3046884208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3046884208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3046884208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3046884208}]},"public_addr":"172.21.15.190:6809/3046884208","cluster_addr":"172.21.15.190:6810/3046884208","heartbeat_back_addr":"172.21.15.190:6812/3046884208","heartbeat_front_addr":"172.21.15.190:6811/3046884208","state":["exists","up"]},{"osd":4,"uuid":"a1aca44b-3dce-4f17-bc06-69bdca9cba04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":693,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":1118550024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1118550024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1118550024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":1118550024}]},"public_addr":"172.21.15.190:6801/1118550024","cluster_addr":"172.21.15.190:6802/1118550024","heartbeat_back_addr":"172.21.15.190:6804/1118550024","heartbeat_front_addr":"172.21.15.190:6803/1118550024","state":["exists","up"]},{"osd":5,"uuid":"87ad0149-d931-4ac6-8787-94f9a9609d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":693,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":189369151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":189369151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":189369151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":189369151}]},"public_addr":"172.21.15.190:6805/189369151","cluster_addr":"172.21.15.190:6806/189369151","heartbeat_back_addr":"172.21.15.190:6808/189369151","heartbeat_front_addr":"172.21.15.190:6807/189369151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.795496+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.359385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:37.802420+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.791133+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.377964+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T05:07:38.428799+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":2,"length":5}]},{"pool":30,"snaps":[{"begin":1,"length":1}]},{"pool":35,"snaps":[{"begin":2,"length":3}]},{"pool":66,"snaps":[{"begin":1,"length":1}]},{"pool":93,"snaps":[{"begin":4,"length":1},{"begin":7,"length":2}]},{"pool":104,"snaps":[{"begin":1,"length":1}]},{"pool":121,"snaps":[{"begin":1,"length":2}]},{"pool":138,"snaps":[{"begin":3,"length":1}]},{"pool":144,"snaps":[{"begin":1,"length":4}]},{"pool":147,"snaps":[{"begin":2,"length":1}]},{"pool":153,"snaps":[{"begin":2,"length":2}]},{"pool":161,"snaps":[{"begin":2,"length":3}]},{"pool":166,"snaps":[{"begin":3,"length":4}]},{"pool":170,"snaps":[{"begin":2,"length":4}]},{"pool":172,"snaps":[{"begin":2,"length":2}]},{"pool":180,"snaps":[{"begin":3,"length":1}]},{"pool":183,"snaps":[{"begin":2,"length":1}]},{"pool":188,"snaps":[{"begin":1,"length":4}]},{"pool":198,"snaps":[{"begin":7,"length":1}]},{"pool":202,"snaps":[{"begin":2,"length":1},{"begin":4,"length":4}]},{"pool":223,"snaps":[{"begin":8,"length":1}]},{"pool":235,"snaps":[{"begin":10,"length":1}]},{"pool":241,"snaps":[{"begin":11,"length":2}]},{"pool":256,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T05:26:36.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 10 librados: watch_flush enter 2024-09-11T05:26:36.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 10 librados: watch_flush exit 2024-09-11T05:26:36.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 20 client.114345.objecter shutdown clearing up homeless session... 2024-09-11T05:26:36.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 10 client.114345.objecter successfully canceled tick 2024-09-11T05:26:36.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdf2803cef0 msgr2=0x7fdf2803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/3060549805 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdf2803cef0 0x7fdf2803f3b0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009a50 tx=0x7fdf38008040 comp rx=0 tx=0).stop 2024-09-11T05:26:36.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 10 monclient: shutdown 2024-09-11T05:26:36.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:36.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf480a1920 msgr2=0x7fdf4813cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:36.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/3060549805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf480a1920 0x7fdf4813cb80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00d800 tx=0x7fdf3c00dcc0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 shutdown_connections 2024-09-11T05:26:36.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/3060549805 >> v2:172.21.15.190:6800/1562782189 conn(0x7fdf2803cef0 0x7fdf2803f3b0 unknown :-1 s=CLOSED pgs=3522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/3060549805 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdf48136170 0x7fdf48137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/3060549805 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdf480a1920 0x7fdf4813cb80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 --2- 172.21.15.138:0/3060549805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf4806e0e0 0x7fdf4813c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:36.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 >> 172.21.15.138:0/3060549805 conn(0x7fdf48063640 msgr2=0x7fdf480a5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:36.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 shutdown_connections 2024-09-11T05:26:36.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 -- 172.21.15.138:0/3060549805 wait complete. 2024-09-11T05:26:36.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:36.655+0000 7fdf4ce43640 1 librados: shutdown 2024-09-11T05:26:36.667 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.667 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308912013}) 2024-09-11T05:26:36.667 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=255594197}) 2024-09-11T05:26:36.667 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.667 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308912013}) 2024-09-11T05:26:36.668 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.671+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=720709469}) 2024-09-11T05:26:36.829 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.831+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.829 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.831+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872994028}) 2024-09-11T05:26:36.829 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.831+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3448468180}) 2024-09-11T05:26:36.829 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.831+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.829 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.831+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872994028}) 2024-09-11T05:26:36.829 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.831+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1001780259}) 2024-09-11T05:26:36.839 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.843+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.839 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.843+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3} crc {logm=1223364741,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2041969564,mon_config_key=174243752}) 2024-09-11T05:26:36.839 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.843+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=73,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4} crc {logm=4116791498,mdsmap=78050798,mgr=1064484177,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=3595442361,mon_config_key=1769693321}) 2024-09-11T05:26:36.839 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.843+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.839 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.843+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3} crc {logm=1223364741,mdsmap=1754867906,mgr=875472461,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2041969564,mon_config_key=174243752}) 2024-09-11T05:26:36.839 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.843+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=71,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5} crc {logm=4034828935,mdsmap=1754867906,mgr=3841245533,mgr_command_descs=2308730518,mgr_metadata=568375954,mgrstat=2041969564,mon_config_key=1194139493}) 2024-09-11T05:26:36.891 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.891+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.891 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=8,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=44} crc {mon_config_key=2387096665,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=1102498740,osd_snap=2097424789,osdmap=3395982931}) 2024-09-11T05:26:36.891 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=8,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=44} crc {mon_config_key=2387096665,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=1102498740,osd_snap=995774523,osdmap=541771453}) 2024-09-11T05:26:36.891 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.891 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=8,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=44} crc {mon_config_key=2387096665,monmap=967756510,nvmeofgw=598337686,osd_metadata=3000249921,osd_pg_creating=1102498740,osd_snap=2097424789,osdmap=3395982931}) 2024-09-11T05:26:36.891 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=7,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=43} crc {mon_config_key=3886154043,monmap=967756510,nvmeofgw=1596038039,osd_metadata=3000249921,osd_pg_creating=1102498740,osd_snap=2587901034,osdmap=1041132391}) 2024-09-11T05:26:36.893 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.893 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2338012858}) 2024-09-11T05:26:36.893 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1872609730}) 2024-09-11T05:26:36.893 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.893 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2338012858}) 2024-09-11T05:26:36.893 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.895+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=365335599}) 2024-09-11T05:26:36.896 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.899+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.896 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.899+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3953386435}) 2024-09-11T05:26:36.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.899+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3291122217}) 2024-09-11T05:26:36.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.899+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.899+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3953386435}) 2024-09-11T05:26:36.897 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.899+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1824166430}) 2024-09-11T05:26:36.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3352151075}) 2024-09-11T05:26:36.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=26} crc {osdmap=3871021856}) 2024-09-11T05:26:36.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T05:26:36.908 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3352151075}) 2024-09-11T05:26:36.909 INFO:tasks.ceph.mon.a.smithi138.stderr:2024-09-11T05:26:36.911+0000 7fe19a4e2640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=580453486}) 2024-09-11T05:26:37.670 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T05:26:37.671 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:26:37.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient: get_monmap_and_config 2024-09-11T05:26:37.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient: build_initial_monmap 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient: monmap: 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:37.810361+0000 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:37.810361+0000 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:37.804 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:37.805 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:37.805 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 1 Processor -- start 2024-09-11T05:26:37.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 1 -- start start 2024-09-11T05:26:37.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient: init 2024-09-11T05:26:37.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:37.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-11T05:26:37.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f647409da20 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:37.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6474074210 0x7f6474074680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient(hunting): picked mon.noname-c con 0x7f6474074210 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f647413b410 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64740a2cf0 con 0x7f647413b410 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64740a2fd0 con 0x7f647409da20 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f647409de00 con 0x7f6474074210 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f647a7aa640 10 monclient(hunting): authenticate will time out at 2081.246649s 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f6478d47640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f647413b410 0x7f647413d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34756/0 (socket says 172.21.15.138:34756) 2024-09-11T05:26:37.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f6478d47640 1 -- 172.21.15.138:0/1215729825 learned_addr learned my addr 172.21.15.138:0/1215729825 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 1 --2- 172.21.15.138:0/1215729825 >> v2:172.21.15.190:3300/0 conn(0x7f6474074210 0x7f6474074680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient(hunting): get_auth_request con 0x7f6474074210 auth_method 0 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient(hunting): handle_auth_done global_id 134122 payload 274 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 1 -- 172.21.15.138:0/1215729825 >> v1:172.21.15.190:6789/0 conn(0x7f647409da20 legacy=0x7f6474132b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:37.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 1 -- 172.21.15.138:0/1215729825 >> v1:172.21.15.138:6789/0 conn(0x7f647413b410 legacy=0x7f647413d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:37.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient: _finish_hunting 0 2024-09-11T05:26:37.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 1 monclient: found mon.noname-c 2024-09-11T05:26:37.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:37.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:26:37.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.807+0000 7f64737fe640 1 -- 172.21.15.138:0/1215729825 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64740a32b0 con 0x7f6474074210 2024-09-11T05:26:37.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f64737fe640 1 -- 172.21.15.138:0/1215729825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6474074210 msgr2=0x7f6474074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:37.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f64737fe640 1 --2- 172.21.15.138:0/1215729825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6474074210 0x7f6474074680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f64640097c0 tx=0x7f646401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c796dd19e4c394bc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 1 -- 172.21.15.138:0/1215729825 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64640b0070 con 0x7f6474074210 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: dump: 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.812 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: _finish_auth 0 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 1 -- 172.21.15.138:0/1215729825 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64640ab600 con 0x7f6474074210 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:37.813 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 1 -- 172.21.15.138:0/1215729825 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64640ab9d0 con 0x7f6474074210 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 5 monclient: authenticate success, global_id 134122 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 4 set_mon_vals no callback set 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6472ffd640 10 monclient: dump: 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.814 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient: shutdown 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- 172.21.15.138:0/1215729825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6474074210 msgr2=0x7f6474074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 --2- 172.21.15.138:0/1215729825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6474074210 0x7f6474074680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f64640097c0 tx=0x7f646401c360 comp rx=0 tx=0).stop 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- 172.21.15.138:0/1215729825 shutdown_connections 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 --2- 172.21.15.138:0/1215729825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6474074210 0x7f6474074680 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- 172.21.15.138:0/1215729825 >> 172.21.15.138:0/1215729825 conn(0x7f64740a7870 msgr2=0x7f64740a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:37.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- 172.21.15.138:0/1215729825 shutdown_connections 2024-09-11T05:26:37.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- 172.21.15.138:0/1215729825 wait complete. 2024-09-11T05:26:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient: build_initial_monmap 2024-09-11T05:26:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient: monmap: 2024-09-11T05:26:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:37.821 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 librados: starting msgr at 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 librados: starting objecter 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 Processor -- start 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- start start 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 librados: setting wanted keys 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 librados: calling monclient init 2024-09-11T05:26:37.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient: init 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6474074210 0x7f64741d0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient(hunting): picked mon.noname-c con 0x7f6474074210 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f647409da20 0x7f64741d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient(hunting): picked mon.noname-b con 0x7f647409da20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f647413b410 0x7f64741d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient(hunting): picked mon.noname-a con 0x7f647413b410 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6474141bb0 con 0x7f647413b410 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6474141a30 con 0x7f647409da20 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6474141d30 con 0x7f6474074210 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f647a7aa640 10 monclient(hunting): authenticate will time out at 2081.255081s 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f647413b410 0x7f64741d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6474074210 0x7f64741d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f647413b410 0x7f64741d0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59618/0 (socket says 172.21.15.138:59618) 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 1 -- 172.21.15.138:0/1816961481 learned_addr learned my addr 172.21.15.138:0/1816961481 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 10 monclient(hunting): get_auth_request con 0x7f647413b410 auth_method 0 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:37.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient(hunting): get_auth_request con 0x7f6474074210 auth_method 0 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f64737fe640 1 --2- 172.21.15.138:0/1816961481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f647409da20 0x7f64741d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient(hunting): handle_auth_done global_id 84647 payload 824 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 1 -- 172.21.15.138:0/1816961481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f647409da20 msgr2=0x7f64741d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 1 --2- 172.21.15.138:0/1816961481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f647409da20 0x7f64741d06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 1 -- 172.21.15.138:0/1816961481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f647413b410 msgr2=0x7f64741d0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 1 --2- 172.21.15.138:0/1816961481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f647413b410 0x7f64741d0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient: _finish_hunting 0 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 1 monclient: found mon.noname-c 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6473fff640 1 -- 172.21.15.138:0/1816961481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647409e1f0 con 0x7f6474074210 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f64737fe640 10 monclient: get_auth_request con 0x7f647409da20 auth_method 0 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f64737fe640 1 --2- 172.21.15.138:0/1816961481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f647409da20 0x7f64741d06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:26:37.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.815+0000 7f6478d47640 1 --2- 172.21.15.138:0/1816961481 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f647413b410 0x7f64741d0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6473fff640 1 -- 172.21.15.138:0/1816961481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6474074210 msgr2=0x7f64741d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6473fff640 1 --2- 172.21.15.138:0/1816961481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6474074210 0x7f64741d0180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f645c00c3e0 tx=0x7f645c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 client.?.objecter ms_handle_connect 0x7f6474074210 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c0119a0 con 0x7f6474074210 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: dump: 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.827 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: _finish_auth 0 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f645c017070 con 0x7f6474074210 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 5 monclient: authenticate success, global_id 84647 2024-09-11T05:26:37.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 10 monclient: _renew_subs 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64741347f0 con 0x7f6474074210 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c011dc0 con 0x7f6474074210 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: dump: 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.829 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f64727fc640 4 set_mon_vals no callback set 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 10 client.84647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 10 monclient: _renew_subs 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6474134510 con 0x7f6474074210 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 1 librados: init done 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 10 client.84647.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:37.830 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 15 client.84647.objecter _session_command_op_assign -1 1 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 20 client.84647.objecter _get_session osd=-1 returning homeless 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 10 client.84647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6474131620 con 0x7f6474074210 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f645c01c460 con 0x7f6474074210 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 --2- 172.21.15.138:0/1816961481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403ced0 0x7f645403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f645c04f500 con 0x7f6474074210 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 client.84647.objecter ms_dispatch 0x7f6474000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 3 client.84647.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 3 client.84647.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 client.84647.objecter checking command 1 2024-09-11T05:26:37.831 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 --2- 172.21.15.138:0/1816961481 >> v2:172.21.15.138:6800/16171034 conn(0x7f64540443a0 0x7f6454046840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 20 client.84647.objecter _get_session s=0x7f6454043cf0 osd=0 3 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 20 client.84647.objecter put_session s=0x7f6454043cf0 osd=0 3 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 15 client.84647.objecter _session_command_op_remove -1 1 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 20 client.84647.objecter _get_session s=0x7f6454043cf0 osd=0 3 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 20 client.84647.objecter get_session s=0x7f6454043cf0 osd=0 3 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 15 client.84647.objecter _session_command_op_assign 0 1 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 20 client.84647.objecter put_session s=0x7f6454043cf0 osd=0 4 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 client.84647.objecter _send_command 1 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 --> v2:172.21.15.138:6800/16171034 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f645c009e50 con 0x7f64540443a0 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 20 client.84647.objecter dump_active .. 0 homeless 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=698) ==== 24+0+0 (secure 0 0 0) 0x7f645c056050 con 0x7f6474074210 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6470ff9640 10 monclient: handle_get_version_reply finishing 1 version 698 2024-09-11T05:26:37.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f64737fe640 1 --2- 172.21.15.138:0/1816961481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403ced0 0x7f645403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6478d47640 1 --2- 172.21.15.138:0/1816961481 >> v2:172.21.15.138:6800/16171034 conn(0x7f64540443a0 0x7f6454046840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f64737fe640 10 monclient: get_auth_request con 0x7f645403ced0 auth_method 0 2024-09-11T05:26:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.819+0000 7f6478d47640 10 monclient: get_auth_request con 0x7f64540443a0 auth_method 0 2024-09-11T05:26:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.823+0000 7f64737fe640 1 --2- 172.21.15.138:0/1816961481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403ced0 0x7f645403f390 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7f64640ab000 tx=0x7f6464008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.823+0000 7f6470ff9640 10 client.84647.objecter ms_handle_connect 0x7f645403ced0 2024-09-11T05:26:37.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.827+0000 7f6470ff9640 10 client.84647.objecter ms_handle_connect 0x7f64540443a0 2024-09-11T05:26:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.827+0000 7f6478d47640 1 --2- 172.21.15.138:0/1816961481 >> v2:172.21.15.138:6800/16171034 conn(0x7f64540443a0 0x7f6454046840 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.827+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 <== osd.0 v2:172.21.15.138:6800/16171034 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f645c009e50 con 0x7f64540443a0 2024-09-11T05:26:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.827+0000 7f6470ff9640 10 client.84647.objecter ms_dispatch 0x7f6474000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:26:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.827+0000 7f6470ff9640 10 client.84647.objecter _finish_command 1 = system:0 2024-09-11T05:26:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.827+0000 7f6470ff9640 20 client.84647.objecter put_session s=0x7f6454043cf0 osd=0 4 2024-09-11T05:26:37.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.827+0000 7f6470ff9640 15 client.84647.objecter _session_command_op_remove 0 1 2024-09-11T05:26:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 10 client.84647.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T05:26:37.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 15 client.84647.objecter _session_command_op_assign -1 2 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 20 client.84647.objecter _get_session s=0x7f6454043cf0 osd=0 3 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 20 client.84647.objecter put_session s=0x7f6454043cf0 osd=0 3 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 20 client.84647.objecter _get_session s=0x7f6454043cf0 osd=0 3 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 15 client.84647.objecter _session_command_op_remove -1 2 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 20 client.84647.objecter get_session s=0x7f6454043cf0 osd=0 3 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 15 client.84647.objecter _session_command_op_assign 0 2 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 20 client.84647.objecter put_session s=0x7f6454043cf0 osd=0 4 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 10 client.84647.objecter _send_command 2 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 --> v2:172.21.15.138:6800/16171034 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6474074700 con 0x7f64540443a0 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f6470ff9640 1 -- 172.21.15.138:0/1816961481 <== osd.0 v2:172.21.15.138:6800/16171034 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6474074700 con 0x7f64540443a0 2024-09-11T05:26:37.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f6470ff9640 10 client.84647.objecter ms_dispatch 0x7f6474000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:26:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f6470ff9640 10 client.84647.objecter _finish_command 2 = system:0 2024-09-11T05:26:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f6470ff9640 20 client.84647.objecter put_session s=0x7f6454043cf0 osd=0 4 2024-09-11T05:26:37.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f6470ff9640 15 client.84647.objecter _session_command_op_remove 0 2 2024-09-11T05:26:37.839 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:37.839 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:26:37.839 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 10 librados: watch_flush enter 2024-09-11T05:26:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 10 librados: watch_flush exit 2024-09-11T05:26:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 10 client.84647.objecter close_session for osd.0 2024-09-11T05:26:37.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 >> v2:172.21.15.138:6800/16171034 conn(0x7f64540443a0 msgr2=0x7f6454046840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 --2- 172.21.15.138:0/1816961481 >> v2:172.21.15.138:6800/16171034 conn(0x7f64540443a0 0x7f6454046840 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 20 client.84647.objecter put_session s=0x7f6454043cf0 osd=0 1 2024-09-11T05:26:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 20 client.84647.objecter shutdown clearing up homeless session... 2024-09-11T05:26:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 10 client.84647.objecter successfully canceled tick 2024-09-11T05:26:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403ced0 msgr2=0x7f645403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 --2- 172.21.15.138:0/1816961481 >> v2:172.21.15.190:6800/1562782189 conn(0x7f645403ced0 0x7f645403f390 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7f64640ab000 tx=0x7f6464008040 comp rx=0 tx=0).stop 2024-09-11T05:26:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 10 monclient: shutdown 2024-09-11T05:26:37.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6474074210 msgr2=0x7f64741d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 --2- 172.21.15.138:0/1816961481 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6474074210 0x7f64741d0180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f645c00c3e0 tx=0x7f645c00c8a0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f6478d47640 1 -- 172.21.15.138:0/1816961481 reap_dead start 2024-09-11T05:26:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 shutdown_connections 2024-09-11T05:26:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.839+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 >> 172.21.15.138:0/1816961481 conn(0x7f64740a7870 msgr2=0x7f647409ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.843+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 shutdown_connections 2024-09-11T05:26:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.843+0000 7f647a7aa640 1 -- 172.21.15.138:0/1816961481 wait complete. 2024-09-11T05:26:37.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.843+0000 7f647a7aa640 1 librados: shutdown 2024-09-11T05:26:37.856 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T05:26:37.970 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient: get_monmap_and_config 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient: build_initial_monmap 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient: monmap: 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:37.977604+0000 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:37.977604+0000 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:37.971 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:37.972 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:37.972 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:37.972 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:37.972 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 1 Processor -- start 2024-09-11T05:26:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 1 -- start start 2024-09-11T05:26:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient: init 2024-09-11T05:26:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-11T05:26:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4d34070010 0x7f4d34135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient(hunting): picked mon.noname-c con 0x7f4d34070010 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:37.975 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d3409e6e0 0x7f4d3409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient(hunting): picked mon.noname-b con 0x7f4d3409e6e0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f4d341382c0 0x7f4d34138770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient(hunting): picked mon.noname-a con 0x7f4d341382c0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f4d340703f0 con 0x7f4d341382c0 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f4d340a8990 con 0x7f4d34070010 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3409cbf0 con 0x7f4d3409e6e0 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d39a1a640 10 monclient(hunting): authenticate will time out at 2081.413665s 2024-09-11T05:26:37.976 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d3409e6e0 0x7f4d3409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f4d341382c0 0x7f4d34138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d3409e6e0 0x7f4d3409eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55026/0 (socket says 172.21.15.138:55026) 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient(hunting): get_auth_request con 0x7f4d341382c0 auth_method 0 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 1 -- 172.21.15.138:0/1241875378 learned_addr learned my addr 172.21.15.138:0/1241875378 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 10 monclient(hunting): get_auth_request con 0x7f4d3409e6e0 auth_method 0 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:37.977 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d32ffd640 1 --2- 172.21.15.138:0/1241875378 >> v2:172.21.15.190:3300/0 conn(0x7f4d34070010 0x7f4d34135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d32ffd640 10 monclient(hunting): get_auth_request con 0x7f4d34070010 auth_method 0 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient(hunting): handle_auth_done global_id 114351 payload 274 2024-09-11T05:26:37.978 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 1 -- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d3409e6e0 msgr2=0x7f4d3409eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 1 --2- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d3409e6e0 0x7f4d3409eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 1 -- 172.21.15.138:0/1241875378 >> v2:172.21.15.190:3300/0 conn(0x7f4d34070010 msgr2=0x7f4d34135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 1 --2- 172.21.15.138:0/1241875378 >> v2:172.21.15.190:3300/0 conn(0x7f4d34070010 0x7f4d34135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient: _finish_hunting 0 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 1 monclient: found mon.noname-a 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d337fe640 1 -- 172.21.15.138:0/1241875378 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3409ced0 con 0x7f4d341382c0 2024-09-11T05:26:37.979 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.975+0000 7f4d327fc640 1 --2- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d3409e6e0 0x7f4d3409eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d337fe640 1 -- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d341382c0 msgr2=0x7f4d34138770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d337fe640 1 --2- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d341382c0 0x7f4d34138770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d3406daa0 tx=0x7f4d2c01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3b866628ff2305 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 1 -- 172.21.15.138:0/1241875378 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0b0070 con 0x7f4d341382c0 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: dump: 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.985 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: _finish_auth 0 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 1 -- 172.21.15.138:0/1241875378 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2c0ab600 con 0x7f4d341382c0 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 1 -- 172.21.15.138:0/1241875378 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0ab9d0 con 0x7f4d341382c0 2024-09-11T05:26:37.986 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 5 monclient: authenticate success, global_id 114351 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 4 set_mon_vals no callback set 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d31ffb640 10 monclient: dump: 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:37.987 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 10 monclient: shutdown 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 -- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d341382c0 msgr2=0x7f4d34138770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d341382c0 0x7f4d34138770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d3406daa0 tx=0x7f4d2c01c310 comp rx=0 tx=0).stop 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 -- 172.21.15.138:0/1241875378 shutdown_connections 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d341382c0 0x7f4d34138770 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/1241875378 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d3409e6e0 0x7f4d3409eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/1241875378 >> v2:172.21.15.190:3300/0 conn(0x7f4d34070010 0x7f4d34135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 -- 172.21.15.138:0/1241875378 >> 172.21.15.138:0/1241875378 conn(0x7f4d340ab830 msgr2=0x7f4d340a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:37.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 -- 172.21.15.138:0/1241875378 shutdown_connections 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.979+0000 7f4d39a1a640 1 -- 172.21.15.138:0/1241875378 wait complete. 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient: build_initial_monmap 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient: monmap: 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 librados: starting msgr at 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 librados: starting objecter 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 Processor -- start 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 -- start start 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 librados: setting wanted keys 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 librados: calling monclient init 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient: init 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d34070010 0x7f4d341d0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient(hunting): picked mon.noname-b con 0x7f4d34070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:37.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d3409e6e0 0x7f4d3413e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient(hunting): picked mon.noname-a con 0x7f4d3409e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d341382c0 0x7f4d3413ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient(hunting): picked mon.noname-c con 0x7f4d341382c0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f4d34141e70 con 0x7f4d3409e6e0 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4d34141ff0 con 0x7f4d34070010 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f4d34142170 con 0x7f4d341382c0 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:37.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d39a1a640 10 monclient(hunting): authenticate will time out at 2081.421321s 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d341382c0 0x7f4d3413ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d3409e6e0 0x7f4d3413e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d341382c0 0x7f4d3413ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55038/0 (socket says 172.21.15.138:55038) 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 -- 172.21.15.138:0/2833219813 learned_addr learned my addr 172.21.15.138:0/2833219813 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient(hunting): get_auth_request con 0x7f4d341382c0 auth_method 0 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 10 monclient(hunting): get_auth_request con 0x7f4d3409e6e0 auth_method 0 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:37.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient(hunting): handle_auth_done global_id 84653 payload 824 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d32ffd640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d34070010 0x7f4d341d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 -- 172.21.15.138:0/2833219813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d34070010 msgr2=0x7f4d341d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d34070010 0x7f4d341d0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 -- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d3409e6e0 msgr2=0x7f4d3413e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d3409e6e0 0x7f4d3413e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:37.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient: _finish_hunting 0 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 monclient: found mon.noname-c 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d337fe640 1 -- 172.21.15.138:0/2833219813 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d340a1f60 con 0x7f4d341382c0 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.983+0000 7f4d327fc640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d3409e6e0 0x7f4d3413e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d337fe640 1 -- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d341382c0 msgr2=0x7f4d3413ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d337fe640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d341382c0 0x7f4d3413ee80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c0ae040 tx=0x7f4d2c0190b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 client.?.objecter ms_handle_connect 0x7f4d341382c0 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:37.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 1 -- 172.21.15.138:0/2833219813 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0b8020 con 0x7f4d341382c0 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: dump: 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:37.995 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: _finish_auth 0 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: _check_auth_tickets 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 1 -- 172.21.15.138:0/2833219813 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2c0b4410 con 0x7f4d341382c0 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 5 monclient: authenticate success, global_id 84653 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 10 monclient: _renew_subs 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3413f3c0 con 0x7f4d341382c0 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 1 -- 172.21.15.138:0/2833219813 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0b0040 con 0x7f4d341382c0 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:37.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 monclient: dump: 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d30ff9640 4 set_mon_vals no callback set 2024-09-11T05:26:37.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 10 client.84653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 10 monclient: _renew_subs 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3413ab50 con 0x7f4d341382c0 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 1 librados: init done 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 1 -- 172.21.15.138:0/2833219813 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f4d2c0b3050 con 0x7f4d341382c0 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d34135450 con 0x7f4d341382c0 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 1 --2- 172.21.15.138:0/2833219813 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4d2003cea0 0x7f4d2003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 1 -- 172.21.15.138:0/2833219813 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f4d2c0e9280 con 0x7f4d341382c0 2024-09-11T05:26:37.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 10 client.84653.objecter ms_dispatch 0x7f4d34000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 3 client.84653.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 3 client.84653.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d1bfff640 20 client.84653.objecter dump_active .. 0 homeless 2024-09-11T05:26:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d32ffd640 1 --2- 172.21.15.138:0/2833219813 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4d2003cea0 0x7f4d2003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.987+0000 7f4d32ffd640 10 monclient: get_auth_request con 0x7f4d2003cea0 auth_method 0 2024-09-11T05:26:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.991+0000 7f4d1bfff640 10 client.84653.objecter ms_handle_connect 0x7f4d2003cea0 2024-09-11T05:26:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.991+0000 7f4d32ffd640 1 --2- 172.21.15.138:0/2833219813 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4d2003cea0 0x7f4d2003f360 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009790 tx=0x7f4d24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:37.999 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.991+0000 7f4d1bfff640 1 -- 172.21.15.138:0/2833219813 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f4d2c0c2030 con 0x7f4d341382c0 2024-09-11T05:26:38.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.991+0000 7f4d1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:38.000 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:37.991+0000 7f4d1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:38.181 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.183+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f4d3413aef0 con 0x7f4d2003cea0 2024-09-11T05:26:38.182 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.183+0000 7f4d1bfff640 1 -- 172.21.15.138:0/2833219813 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4d3413aef0 con 0x7f4d2003cea0 2024-09-11T05:26:38.182 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T05:26:38.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 10 librados: watch_flush enter 2024-09-11T05:26:38.184 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 10 librados: watch_flush exit 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 20 client.84653.objecter shutdown clearing up homeless session... 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 10 client.84653.objecter successfully canceled tick 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4d2003cea0 msgr2=0x7f4d2003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/2833219813 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4d2003cea0 0x7f4d2003f360 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009790 tx=0x7f4d24008040 comp rx=0 tx=0).stop 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 10 monclient: shutdown 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d341382c0 msgr2=0x7f4d3413ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d341382c0 0x7f4d3413ee80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c0ae040 tx=0x7f4d2c0190b0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 shutdown_connections 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/2833219813 >> v2:172.21.15.190:6800/1562782189 conn(0x7f4d2003cea0 0x7f4d2003f360 unknown :-1 s=CLOSED pgs=3524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.185 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f4d341382c0 0x7f4d3413ee80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f4d3409e6e0 0x7f4d3413e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 --2- 172.21.15.138:0/2833219813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d34070010 0x7f4d341d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 >> 172.21.15.138:0/2833219813 conn(0x7f4d340ab830 msgr2=0x7f4d340a8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:38.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 shutdown_connections 2024-09-11T05:26:38.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 -- 172.21.15.138:0/2833219813 wait complete. 2024-09-11T05:26:38.186 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.187+0000 7f4d39a1a640 1 librados: shutdown 2024-09-11T05:26:38.201 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T05:26:38.201 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:26:38.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient: get_monmap_and_config 2024-09-11T05:26:38.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient: build_initial_monmap 2024-09-11T05:26:38.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient: monmap: 2024-09-11T05:26:38.327 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:38.334079+0000 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:38.334079+0000 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:38.328 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 1 Processor -- start 2024-09-11T05:26:38.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 1 -- start start 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient: init 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdeb00a5cc0 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdeb00a67b0 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdeb013f1d0 0x7fdeb01415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb013f1d0 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb0070700 con 0x7fdeb00a67b0 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb00a2a70 con 0x7fdeb00a5cc0 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdeb00a2d50 con 0x7fdeb013f1d0 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb6a81640 10 monclient(hunting): authenticate will time out at 2081.770482s 2024-09-11T05:26:38.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeaffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7fdeb00a67b0 0x7fdeb013dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34768/0 (socket says 172.21.15.138:34768) 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeaffff640 1 -- 172.21.15.138:0/652755846 learned_addr learned my addr 172.21.15.138:0/652755846 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 1 --2- 172.21.15.138:0/652755846 >> v2:172.21.15.190:3300/0 conn(0x7fdeb013f1d0 0x7fdeb01415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient(hunting): get_auth_request con 0x7fdeb013f1d0 auth_method 0 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient(hunting): handle_auth_done global_id 134128 payload 274 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 1 -- 172.21.15.138:0/652755846 >> v1:172.21.15.190:6789/0 conn(0x7fdeb00a5cc0 legacy=0x7fdeb00a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 1 -- 172.21.15.138:0/652755846 >> v1:172.21.15.138:6789/0 conn(0x7fdeb00a67b0 legacy=0x7fdeb013dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient: _finish_hunting 0 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 1 monclient: found mon.noname-c 2024-09-11T05:26:38.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:38.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:26:38.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.331+0000 7fdeb501e640 1 -- 172.21.15.138:0/652755846 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb00a3030 con 0x7fdeb013f1d0 2024-09-11T05:26:38.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.335+0000 7fdeb501e640 1 -- 172.21.15.138:0/652755846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb013f1d0 msgr2=0x7fdeb01415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:38.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.335+0000 7fdeb501e640 1 --2- 172.21.15.138:0/652755846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb013f1d0 0x7fdeb01415c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009dc0 tx=0x7fdea801c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=338a6d663bffcb4c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 1 -- 172.21.15.138:0/652755846 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea80b2070 con 0x7fdeb013f1d0 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: dump: 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:38.344 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: _finish_auth 0 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 5 monclient: authenticate success, global_id 134128 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 1 -- 172.21.15.138:0/652755846 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea80a58f0 con 0x7fdeb013f1d0 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:38.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 4 set_mon_vals no callback set 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 1 -- 172.21.15.138:0/652755846 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea80a5cc0 con 0x7fdeb013f1d0 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaf7fe640 10 monclient: dump: 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:38.346 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient: shutdown 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- 172.21.15.138:0/652755846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb013f1d0 msgr2=0x7fdeb01415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 --2- 172.21.15.138:0/652755846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb013f1d0 0x7fdeb01415c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009dc0 tx=0x7fdea801c810 comp rx=0 tx=0).stop 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- 172.21.15.138:0/652755846 shutdown_connections 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 --2- 172.21.15.138:0/652755846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb013f1d0 0x7fdeb01415c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- 172.21.15.138:0/652755846 >> 172.21.15.138:0/652755846 conn(0x7fdeb00a89b0 msgr2=0x7fdeb0076910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- 172.21.15.138:0/652755846 shutdown_connections 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- 172.21.15.138:0/652755846 wait complete. 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient: build_initial_monmap 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient: monmap: 2024-09-11T05:26:38.347 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 librados: starting msgr at 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 librados: starting objecter 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 Processor -- start 2024-09-11T05:26:38.348 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- start start 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 librados: setting wanted keys 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 librados: calling monclient init 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient: init 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb00a5cc0 0x7fdeb01d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient(hunting): picked mon.noname-b con 0x7fdeb00a5cc0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb00a67b0 0x7fdeb01d1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb00a67b0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb013f1d0 0x7fdeb013bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient(hunting): picked mon.noname-a con 0x7fdeb013f1d0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb0070460 con 0x7fdeb013f1d0 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.349 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb00702e0 con 0x7fdeb00a5cc0 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb0144730 con 0x7fdeb00a67b0 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb6a81640 10 monclient(hunting): authenticate will time out at 2081.778741s 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb013f1d0 0x7fdeb013bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb00a67b0 0x7fdeb01d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb00a67b0 0x7fdeb01d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55044/0 (socket says 172.21.15.138:55044) 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 1 -- 172.21.15.138:0/3384254212 learned_addr learned my addr 172.21.15.138:0/3384254212 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb013f1d0 0x7fdeb013bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59646/0 (socket says 172.21.15.138:59646) 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 10 monclient(hunting): get_auth_request con 0x7fdeb00a67b0 auth_method 0 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.350 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient(hunting): get_auth_request con 0x7fdeb013f1d0 auth_method 0 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb481d640 1 --2- 172.21.15.138:0/3384254212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb00a5cc0 0x7fdeb01d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb481d640 10 monclient(hunting): get_auth_request con 0x7fdeb00a5cc0 auth_method 0 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb481d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.351 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb481d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb481d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient(hunting): handle_auth_done global_id 114357 payload 824 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 1 -- 172.21.15.138:0/3384254212 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb00a67b0 msgr2=0x7fdeb01d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 1 --2- 172.21.15.138:0/3384254212 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb00a67b0 0x7fdeb01d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 1 -- 172.21.15.138:0/3384254212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb00a5cc0 msgr2=0x7fdeb01d0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 1 --2- 172.21.15.138:0/3384254212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdeb00a5cc0 0x7fdeb01d0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient: _finish_hunting 0 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 1 monclient: found mon.noname-a 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeb501e640 1 -- 172.21.15.138:0/3384254212 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb0122320 con 0x7fdeb013f1d0 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.339+0000 7fdeaffff640 1 --2- 172.21.15.138:0/3384254212 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fdeb00a67b0 0x7fdeb01d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:38.352 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb501e640 1 -- 172.21.15.138:0/3384254212 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb013f1d0 msgr2=0x7fdeb013bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb501e640 1 --2- 172.21.15.138:0/3384254212 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb013f1d0 0x7fdeb013bd50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdea80ac000 tx=0x7fdea80195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 client.?.objecter ms_handle_connect 0x7fdeb013f1d0 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea80bd030 con 0x7fdeb013f1d0 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: dump: 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.353 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: _finish_auth 0 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea80b8680 con 0x7fdeb013f1d0 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 5 monclient: authenticate success, global_id 114357 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 10 monclient: _renew_subs 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb00a24b0 con 0x7fdeb013f1d0 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea80b2040 con 0x7fdeb013f1d0 2024-09-11T05:26:38.354 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: dump: 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 10 client.114357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:38.355 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 10 monclient: _renew_subs 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeaeffd640 4 set_mon_vals no callback set 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb013ea70 con 0x7fdeb013f1d0 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 1 librados: init done 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 10 client.114357.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 15 client.114357.objecter _session_command_op_assign -1 1 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 20 client.114357.objecter _get_session osd=-1 returning homeless 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 10 client.114357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdeb0122160 con 0x7fdeb013f1d0 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fdea80ca060 con 0x7fdeb013f1d0 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 --2- 172.21.15.138:0/3384254212 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9003cea0 0x7fde9003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7fdea80ea4d0 con 0x7fdeb013f1d0 2024-09-11T05:26:38.356 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 client.114357.objecter ms_dispatch 0x7fdeb0000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 3 client.114357.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 3 client.114357.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 client.114357.objecter checking command 1 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 --2- 172.21.15.138:0/3384254212 >> v2:172.21.15.138:6808/4002974512 conn(0x7fde90044380 0x7fde90046820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 20 client.114357.objecter _get_session s=0x7fde90043cd0 osd=1 3 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 20 client.114357.objecter put_session s=0x7fde90043cd0 osd=1 3 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 15 client.114357.objecter _session_command_op_remove -1 1 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 20 client.114357.objecter _get_session s=0x7fde90043cd0 osd=1 3 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 20 client.114357.objecter get_session s=0x7fde90043cd0 osd=1 3 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 15 client.114357.objecter _session_command_op_assign 1 1 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 20 client.114357.objecter put_session s=0x7fde90043cd0 osd=1 4 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 client.114357.objecter _send_command 1 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 --> v2:172.21.15.138:6808/4002974512 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdea80c5e70 con 0x7fde90044380 2024-09-11T05:26:38.357 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 20 client.114357.objecter dump_active .. 0 homeless 2024-09-11T05:26:38.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_get_version_reply(handle=1 version=698) ==== 24+0+0 (secure 0 0 0) 0x7fdea80f1050 con 0x7fdeb013f1d0 2024-09-11T05:26:38.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdead7fa640 10 monclient: handle_get_version_reply finishing 1 version 698 2024-09-11T05:26:38.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeaffff640 1 --2- 172.21.15.138:0/3384254212 >> v2:172.21.15.138:6808/4002974512 conn(0x7fde90044380 0x7fde90046820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeaffff640 10 monclient: get_auth_request con 0x7fde90044380 auth_method 0 2024-09-11T05:26:38.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb481d640 1 --2- 172.21.15.138:0/3384254212 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9003cea0 0x7fde9003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.358 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.343+0000 7fdeb481d640 10 monclient: get_auth_request con 0x7fde9003cea0 auth_method 0 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.347+0000 7fdead7fa640 10 client.114357.objecter ms_handle_connect 0x7fde9003cea0 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.347+0000 7fdeb481d640 1 --2- 172.21.15.138:0/3384254212 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9003cea0 0x7fde9003f360 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7fdea0004e30 tx=0x7fdea0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.347+0000 7fdead7fa640 10 client.114357.objecter ms_handle_connect 0x7fde90044380 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.347+0000 7fdeaffff640 1 --2- 172.21.15.138:0/3384254212 >> v2:172.21.15.138:6808/4002974512 conn(0x7fde90044380 0x7fde90046820 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.351+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 <== osd.1 v2:172.21.15.138:6808/4002974512 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdea80c5e70 con 0x7fde90044380 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.351+0000 7fdead7fa640 10 client.114357.objecter ms_dispatch 0x7fdeb0000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.351+0000 7fdead7fa640 10 client.114357.objecter _finish_command 1 = system:0 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.351+0000 7fdead7fa640 20 client.114357.objecter put_session s=0x7fde90043cd0 osd=1 4 2024-09-11T05:26:38.359 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.351+0000 7fdead7fa640 15 client.114357.objecter _session_command_op_remove 1 1 2024-09-11T05:26:38.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 10 client.114357.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T05:26:38.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 15 client.114357.objecter _session_command_op_assign -1 2 2024-09-11T05:26:38.360 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 20 client.114357.objecter _get_session s=0x7fde90043cd0 osd=1 3 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 20 client.114357.objecter put_session s=0x7fde90043cd0 osd=1 3 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 20 client.114357.objecter _get_session s=0x7fde90043cd0 osd=1 3 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 15 client.114357.objecter _session_command_op_remove -1 2 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 20 client.114357.objecter get_session s=0x7fde90043cd0 osd=1 3 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 15 client.114357.objecter _session_command_op_assign 1 2 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 20 client.114357.objecter put_session s=0x7fde90043cd0 osd=1 4 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 10 client.114357.objecter _send_command 2 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 --> v2:172.21.15.138:6808/4002974512 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdeb00a60a0 con 0x7fde90044380 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdead7fa640 1 -- 172.21.15.138:0/3384254212 <== osd.1 v2:172.21.15.138:6808/4002974512 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdeb00a60a0 con 0x7fde90044380 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdead7fa640 10 client.114357.objecter ms_dispatch 0x7fdeb0000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdead7fa640 10 client.114357.objecter _finish_command 2 = system:0 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdead7fa640 20 client.114357.objecter put_session s=0x7fde90043cd0 osd=1 4 2024-09-11T05:26:38.361 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdead7fa640 15 client.114357.objecter _session_command_op_remove 1 2 2024-09-11T05:26:38.362 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:38.362 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:26:38.362 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:38.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 10 librados: watch_flush enter 2024-09-11T05:26:38.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 10 librados: watch_flush exit 2024-09-11T05:26:38.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 10 client.114357.objecter close_session for osd.1 2024-09-11T05:26:38.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 >> v2:172.21.15.138:6808/4002974512 conn(0x7fde90044380 msgr2=0x7fde90046820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 --2- 172.21.15.138:0/3384254212 >> v2:172.21.15.138:6808/4002974512 conn(0x7fde90044380 0x7fde90046820 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 20 client.114357.objecter put_session s=0x7fde90043cd0 osd=1 1 2024-09-11T05:26:38.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 20 client.114357.objecter shutdown clearing up homeless session... 2024-09-11T05:26:38.363 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 10 client.114357.objecter successfully canceled tick 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9003cea0 msgr2=0x7fde9003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 --2- 172.21.15.138:0/3384254212 >> v2:172.21.15.190:6800/1562782189 conn(0x7fde9003cea0 0x7fde9003f360 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7fdea0004e30 tx=0x7fdea0009040 comp rx=0 tx=0).stop 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 10 monclient: shutdown 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb013f1d0 msgr2=0x7fdeb013bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 --2- 172.21.15.138:0/3384254212 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fdeb013f1d0 0x7fdeb013bd50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdea80ac000 tx=0x7fdea80195b0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb501e640 1 -- 172.21.15.138:0/3384254212 reap_dead start 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 shutdown_connections 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 >> 172.21.15.138:0/3384254212 conn(0x7fdeb00a89b0 msgr2=0x7fdeb013fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 shutdown_connections 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 -- 172.21.15.138:0/3384254212 wait complete. 2024-09-11T05:26:38.364 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.363+0000 7fdeb6a81640 1 librados: shutdown 2024-09-11T05:26:38.377 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T05:26:38.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient: get_monmap_and_config 2024-09-11T05:26:38.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient: build_initial_monmap 2024-09-11T05:26:38.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient: monmap: 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:38.498094+0000 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:38.498094+0000 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:38.492 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 1 Processor -- start 2024-09-11T05:26:38.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 1 -- start start 2024-09-11T05:26:38.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient: init 2024-09-11T05:26:38.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f78ec09da20 0x7f78ec0a1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient(hunting): picked mon.noname-c con 0x7f78ec09da20 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec0a1e60 0x7f78ec0a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient(hunting): picked mon.noname-b con 0x7f78ec0a1e60 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f78ec136200 0x7f78ec136650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient(hunting): picked mon.noname-a con 0x7f78ec136200 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f78ec09de00 con 0x7f78ec136200 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f78ec064590 con 0x7f78ec09da20 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f78ec0a2c70 con 0x7f78ec0a1e60 2024-09-11T05:26:38.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f2677640 10 monclient(hunting): authenticate will time out at 2081.934138s 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec0a1e60 0x7f78ec0a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f78ec136200 0x7f78ec136650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f78ec136200 0x7f78ec136650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59658/0 (socket says 172.21.15.138:59658) 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec0a1e60 0x7f78ec0a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55048/0 (socket says 172.21.15.138:55048) 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 -- 172.21.15.138:0/3827049451 learned_addr learned my addr 172.21.15.138:0/3827049451 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient(hunting): get_auth_request con 0x7f78ec136200 auth_method 0 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 10 monclient(hunting): get_auth_request con 0x7f78ec0a1e60 auth_method 0 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78ebfff640 1 --2- 172.21.15.138:0/3827049451 >> v2:172.21.15.190:3300/0 conn(0x7f78ec09da20 0x7f78ec0a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78ebfff640 10 monclient(hunting): get_auth_request con 0x7f78ec09da20 auth_method 0 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient(hunting): handle_auth_done global_id 114360 payload 274 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 -- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec0a1e60 msgr2=0x7f78ec0a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:38.497 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 --2- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec0a1e60 0x7f78ec0a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 -- 172.21.15.138:0/3827049451 >> v2:172.21.15.190:3300/0 conn(0x7f78ec09da20 msgr2=0x7f78ec0a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:38.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 --2- 172.21.15.138:0/3827049451 >> v2:172.21.15.190:3300/0 conn(0x7f78ec09da20 0x7f78ec0a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient: _finish_hunting 0 2024-09-11T05:26:38.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 monclient: found mon.noname-a 2024-09-11T05:26:38.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:38.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:38.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78f0c14640 1 -- 172.21.15.138:0/3827049451 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ec0a2f50 con 0x7f78ec136200 2024-09-11T05:26:38.498 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.495+0000 7f78eb7fe640 1 --2- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec0a1e60 0x7f78ec0a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f0c14640 1 -- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec136200 msgr2=0x7f78ec136650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f0c14640 1 --2- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec136200 0x7f78ec136650 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f78ec06d670 tx=0x7f78e401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998a6a433665ab0d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 1 -- 172.21.15.138:0/3827049451 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40a5640 con 0x7f78ec136200 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: dump: 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.503 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: _finish_auth 0 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: _check_auth_tickets 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 5 monclient: authenticate success, global_id 114360 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 1 -- 172.21.15.138:0/3827049451 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e40a57e0 con 0x7f78ec136200 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:38.504 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 4 set_mon_vals no callback set 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 1 -- 172.21.15.138:0/3827049451 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40a5bb0 con 0x7f78ec136200 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78eaffd640 10 monclient: dump: 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:38.505 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 10 monclient: shutdown 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 -- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec136200 msgr2=0x7f78ec136650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 --2- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec136200 0x7f78ec136650 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f78ec06d670 tx=0x7f78e401c8b0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 -- 172.21.15.138:0/3827049451 shutdown_connections 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 --2- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec136200 0x7f78ec136650 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 --2- 172.21.15.138:0/3827049451 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec0a1e60 0x7f78ec0a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 --2- 172.21.15.138:0/3827049451 >> v2:172.21.15.190:3300/0 conn(0x7f78ec09da20 0x7f78ec0a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 -- 172.21.15.138:0/3827049451 >> 172.21.15.138:0/3827049451 conn(0x7f78ec063640 msgr2=0x7f78ec063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 -- 172.21.15.138:0/3827049451 shutdown_connections 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.499+0000 7f78f2677640 1 -- 172.21.15.138:0/3827049451 wait complete. 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient: build_initial_monmap 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient: monmap: 2024-09-11T05:26:38.506 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 librados: starting msgr at 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 librados: starting objecter 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 Processor -- start 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 -- start start 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 librados: setting wanted keys 2024-09-11T05:26:38.507 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 librados: calling monclient init 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient: init 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78ec09da20 0x7f78ec12ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient(hunting): picked mon.noname-b con 0x7f78ec09da20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec0a1e60 0x7f78ec12f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient(hunting): picked mon.noname-a con 0x7f78ec0a1e60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec136200 0x7f78ec1cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient(hunting): picked mon.noname-c con 0x7f78ec136200 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec13fed0 con 0x7f78ec0a1e60 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec13fd50 con 0x7f78ec09da20 2024-09-11T05:26:38.508 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f78ec140050 con 0x7f78ec136200 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f2677640 10 monclient(hunting): authenticate will time out at 2081.941258s 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec136200 0x7f78ec1cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec0a1e60 0x7f78ec12f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec0a1e60 0x7f78ec12f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59666/0 (socket says 172.21.15.138:59666) 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 1 -- 172.21.15.138:0/2459083953 learned_addr learned my addr 172.21.15.138:0/2459083953 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 10 monclient(hunting): get_auth_request con 0x7f78ec0a1e60 auth_method 0 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient(hunting): get_auth_request con 0x7f78ec136200 auth_method 0 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.509 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78ebfff640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78ec09da20 0x7f78ec12ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78ebfff640 10 monclient(hunting): get_auth_request con 0x7f78ec09da20 auth_method 0 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:38.510 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient(hunting): handle_auth_done global_id 84662 payload 824 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 1 -- 172.21.15.138:0/2459083953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78ec09da20 msgr2=0x7f78ec12ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78ec09da20 0x7f78ec12ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 1 -- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec0a1e60 msgr2=0x7f78ec12f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec0a1e60 0x7f78ec12f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient: _finish_hunting 0 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 1 monclient: found mon.noname-c 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78f0c14640 1 -- 172.21.15.138:0/2459083953 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ec135d40 con 0x7f78ec136200 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.503+0000 7f78eb7fe640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec0a1e60 0x7f78ec12f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f0c14640 1 -- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec136200 msgr2=0x7f78ec1cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f0c14640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec136200 0x7f78ec1cd060 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0a6e50 tx=0x7f78e4019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 client.?.objecter ms_handle_connect 0x7f78ec136200 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:38.511 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 1 -- 172.21.15.138:0/2459083953 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40a5640 con 0x7f78ec136200 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: dump: 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: _finish_auth 0 2024-09-11T05:26:38.512 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 5 monclient: authenticate success, global_id 84662 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 1 -- 172.21.15.138:0/2459083953 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e40b6690 con 0x7f78ec136200 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 10 monclient: _renew_subs 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ec09f600 con 0x7f78ec136200 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 1 -- 172.21.15.138:0/2459083953 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40b6a80 con 0x7f78ec136200 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 10 monclient: dump: 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:38.513 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78ea7fc640 4 set_mon_vals no callback set 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 10 client.84662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 10 monclient: _renew_subs 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ec09f320 con 0x7f78ec136200 2024-09-11T05:26:38.514 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 1 librados: init done 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ec0a22a0 con 0x7f78ec136200 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78f0c14640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec136200 0x7f78ec1cd060 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0a6e50 tx=0x7f78e4019650 comp rx=0 tx=0).queue_received will delay after 1.72603e+09 on 0x7f78e40e9bb0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 1 -- 172.21.15.138:0/2459083953 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f78e40b6c20 con 0x7f78ec136200 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78e8ff9640 1 --2- 172.21.15.138:0/2459083953 >> v2:172.21.15.190:6800/1562782189 conn(0x7f78d803cea0 0x7f78d803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78ebfff640 1 --2- 172.21.15.138:0/2459083953 >> v2:172.21.15.190:6800/1562782189 conn(0x7f78d803cea0 0x7f78d803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.507+0000 7f78ebfff640 10 monclient: get_auth_request con 0x7f78d803cea0 auth_method 0 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.511+0000 7f78ebfff640 1 --2- 172.21.15.138:0/2459083953 >> v2:172.21.15.190:6800/1562782189 conn(0x7f78d803cea0 0x7f78d803f360 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f78ec09ef70 tx=0x7f78dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:38.516 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.511+0000 7f78e8ff9640 10 client.84662.objecter ms_handle_connect 0x7f78d803cea0 2024-09-11T05:26:38.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.511+0000 7f78e8ff9640 1 -- 172.21.15.138:0/2459083953 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f78e40e9bb0 con 0x7f78ec136200 2024-09-11T05:26:38.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.511+0000 7f78e8ff9640 10 client.84662.objecter ms_dispatch 0x7f78ec000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:38.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.511+0000 7f78e8ff9640 3 client.84662.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:38.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.511+0000 7f78e8ff9640 3 client.84662.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:38.517 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:38.511+0000 7f78e8ff9640 20 client.84662.objecter dump_active .. 0 homeless 2024-09-11T05:26:39.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.335+0000 7f78e8ff9640 1 -- 172.21.15.138:0/2459083953 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f78e40c5020 con 0x7f78ec136200 2024-09-11T05:26:39.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.335+0000 7f78e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:39.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.335+0000 7f78e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:39.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.503+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f78ec09e110 con 0x7f78d803cea0 2024-09-11T05:26:39.501 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.503+0000 7f78e8ff9640 1 -- 172.21.15.138:0/2459083953 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f78ec09e110 con 0x7f78d803cea0 2024-09-11T05:26:39.502 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T05:26:39.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 10 librados: watch_flush enter 2024-09-11T05:26:39.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 10 librados: watch_flush exit 2024-09-11T05:26:39.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 20 client.84662.objecter shutdown clearing up homeless session... 2024-09-11T05:26:39.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 10 client.84662.objecter successfully canceled tick 2024-09-11T05:26:39.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 >> v2:172.21.15.190:6800/1562782189 conn(0x7f78d803cea0 msgr2=0x7f78d803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 --2- 172.21.15.138:0/2459083953 >> v2:172.21.15.190:6800/1562782189 conn(0x7f78d803cea0 0x7f78d803f360 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f78ec09ef70 tx=0x7f78dc008040 comp rx=0 tx=0).stop 2024-09-11T05:26:39.505 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 10 monclient: shutdown 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec136200 msgr2=0x7f78ec1cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec136200 0x7f78ec1cd060 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0a6e50 tx=0x7f78e4019650 comp rx=0 tx=0).stop 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 shutdown_connections 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 --2- 172.21.15.138:0/2459083953 >> v2:172.21.15.190:6800/1562782189 conn(0x7f78d803cea0 0x7f78d803f360 unknown :-1 s=CLOSED pgs=3526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f78ec136200 0x7f78ec1cd060 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f78ec0a1e60 0x7f78ec12f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 --2- 172.21.15.138:0/2459083953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f78ec09da20 0x7f78ec12ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 >> 172.21.15.138:0/2459083953 conn(0x7f78ec063640 msgr2=0x7f78ec12fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 shutdown_connections 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 -- 172.21.15.138:0/2459083953 wait complete. 2024-09-11T05:26:39.506 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.507+0000 7f78f2677640 1 librados: shutdown 2024-09-11T05:26:39.520 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T05:26:39.520 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:26:39.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient: get_monmap_and_config 2024-09-11T05:26:39.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient: build_initial_monmap 2024-09-11T05:26:39.643 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient: monmap: 2024-09-11T05:26:39.643 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:39.650047+0000 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:39.650047+0000 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:39.644 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 1 Processor -- start 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 1 -- start start 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient: init 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809da20 0x7fcb0809e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0809da20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcb080a07c0 0x7fcb0809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient(hunting): picked mon.noname-c con 0x7fcb080a07c0 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb0809ef80 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:39.647 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb080a42c0 con 0x7fcb0809ef80 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fcb080a14e0 con 0x7fcb080a07c0 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0809fe40 con 0x7fcb0809da20 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb0e76a640 10 monclient(hunting): authenticate will time out at 2083.086503s 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809da20 0x7fcb0809e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809da20 0x7fcb0809e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55078/0 (socket says 172.21.15.138:55078) 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 1 -- 172.21.15.138:0/180573581 learned_addr learned my addr 172.21.15.138:0/180573581 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient(hunting): get_auth_request con 0x7fcb0809da20 auth_method 0 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:39.648 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient(hunting): handle_auth_done global_id 84665 payload 274 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb077fe640 1 --2- 172.21.15.138:0/180573581 >> v2:172.21.15.190:3300/0 conn(0x7fcb080a07c0 0x7fcb0809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 1 -- 172.21.15.138:0/180573581 >> v2:172.21.15.190:3300/0 conn(0x7fcb080a07c0 msgr2=0x7fcb0809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 1 --2- 172.21.15.138:0/180573581 >> v2:172.21.15.190:3300/0 conn(0x7fcb080a07c0 0x7fcb0809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 1 -- 172.21.15.138:0/180573581 >> v1:172.21.15.190:6789/0 conn(0x7fcb0809ef80 legacy=0x7fcb0813b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient: _finish_hunting 0 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 1 monclient: found mon.noname-b 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:39.649 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:26:39.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb07fff640 1 -- 172.21.15.138:0/180573581 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb080a1210 con 0x7fcb0809da20 2024-09-11T05:26:39.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb077fe640 10 monclient: get_auth_request con 0x7fcb080a07c0 auth_method 0 2024-09-11T05:26:39.650 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.647+0000 7fcb077fe640 1 --2- 172.21.15.138:0/180573581 >> v2:172.21.15.190:3300/0 conn(0x7fcb080a07c0 0x7fcb0809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb07fff640 1 -- 172.21.15.138:0/180573581 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809da20 msgr2=0x7fcb0809e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb07fff640 1 --2- 172.21.15.138:0/180573581 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809da20 0x7fcb0809e500 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009870 tx=0x7fcaf801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c9ca161abdd9465 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 1 -- 172.21.15.138:0/180573581 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80b0070 con 0x7fcb0809da20 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: dump: 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.654 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: _finish_auth 0 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: _check_auth_tickets 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb0e76a640 5 monclient: authenticate success, global_id 84665 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb0e76a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 1 -- 172.21.15.138:0/180573581 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf80ab600 con 0x7fcb0809da20 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 1 -- 172.21.15.138:0/180573581 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80ab9d0 con 0x7fcb0809da20 2024-09-11T05:26:39.655 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb06ffd640 10 monclient: dump: 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb0e76a640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.651+0000 7fcb0e76a640 4 set_mon_vals no callback set 2024-09-11T05:26:39.656 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient: shutdown 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- 172.21.15.138:0/180573581 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809da20 msgr2=0x7fcb0809e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 --2- 172.21.15.138:0/180573581 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809da20 0x7fcb0809e500 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009870 tx=0x7fcaf801c420 comp rx=0 tx=0).stop 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- 172.21.15.138:0/180573581 shutdown_connections 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 --2- 172.21.15.138:0/180573581 >> v2:172.21.15.190:3300/0 conn(0x7fcb080a07c0 0x7fcb0809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 --2- 172.21.15.138:0/180573581 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809da20 0x7fcb0809e500 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- 172.21.15.138:0/180573581 >> 172.21.15.138:0/180573581 conn(0x7fcb080a7870 msgr2=0x7fcb080a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- 172.21.15.138:0/180573581 shutdown_connections 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- 172.21.15.138:0/180573581 wait complete. 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient: build_initial_monmap 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient: monmap: 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:39.657 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 librados: starting msgr at 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 librados: starting objecter 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 Processor -- start 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- start start 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 librados: setting wanted keys 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 librados: calling monclient init 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient: init 2024-09-11T05:26:39.658 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809ef80 0x7fcb0806fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient(hunting): picked mon.noname-c con 0x7fcb0809ef80 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb081d0aa0 0x7fcb08133fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient(hunting): picked mon.noname-b con 0x7fcb081d0aa0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb081cd0c0 0x7fcb081cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient(hunting): picked mon.noname-a con 0x7fcb081cd0c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0813fb70 con 0x7fcb081cd0c0 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0813f9f0 con 0x7fcb081d0aa0 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0813fcf0 con 0x7fcb0809ef80 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:39.659 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0e76a640 10 monclient(hunting): authenticate will time out at 2083.093876s 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809ef80 0x7fcb0806fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809ef80 0x7fcb0806fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55084/0 (socket says 172.21.15.138:55084) 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 -- 172.21.15.138:0/2829919590 learned_addr learned my addr 172.21.15.138:0/2829919590 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient(hunting): get_auth_request con 0x7fcb0809ef80 auth_method 0 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0cd07640 1 --2- 172.21.15.138:0/2829919590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb081cd0c0 0x7fcb081cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0cd07640 10 monclient(hunting): get_auth_request con 0x7fcb081cd0c0 auth_method 0 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0cd07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0cd07640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0cd07640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:39.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0cd07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0cd07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb0cd07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb077fe640 1 --2- 172.21.15.138:0/2829919590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb081d0aa0 0x7fcb08133fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb077fe640 10 monclient(hunting): get_auth_request con 0x7fcb081d0aa0 auth_method 0 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient(hunting): handle_auth_done global_id 84668 payload 824 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 -- 172.21.15.138:0/2829919590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb081d0aa0 msgr2=0x7fcb08133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 --2- 172.21.15.138:0/2829919590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb081d0aa0 0x7fcb08133fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 -- 172.21.15.138:0/2829919590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb081cd0c0 msgr2=0x7fcb081cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 --2- 172.21.15.138:0/2829919590 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7fcb081cd0c0 0x7fcb081cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient: _finish_hunting 0 2024-09-11T05:26:39.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 monclient: found mon.noname-c 2024-09-11T05:26:39.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:39.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:39.662 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.655+0000 7fcb07fff640 1 -- 172.21.15.138:0/2829919590 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb080a5f60 con 0x7fcb0809ef80 2024-09-11T05:26:39.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb07fff640 1 -- 172.21.15.138:0/2829919590 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809ef80 msgr2=0x7fcb0806fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:39.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb07fff640 1 --2- 172.21.15.138:0/2829919590 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809ef80 0x7fcb0806fb30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcaf801c250 tx=0x7fcaf80191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 client.?.objecter ms_handle_connect 0x7fcb0809ef80 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80b0040 con 0x7fcb0809ef80 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: dump: 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.665 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: _finish_auth 0 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: _check_auth_tickets 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 5 monclient: authenticate success, global_id 84668 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 10 monclient: _renew_subs 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf80aa5b0 con 0x7fcb0809ef80 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0813c670 con 0x7fcb0809ef80 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80aa9a0 con 0x7fcb0809ef80 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:39.666 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: dump: 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 10 client.84668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 10 monclient: _renew_subs 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:39.667 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb067fc640 4 set_mon_vals no callback set 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0813cc90 con 0x7fcb0809ef80 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 1 librados: init done 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 10 client.84668.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 15 client.84668.objecter _session_command_op_assign -1 1 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 20 client.84668.objecter _get_session osd=-1 returning homeless 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 10 client.84668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7fcaf80aab80 con 0x7fcb0809ef80 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb0809e500 con 0x7fcb0809ef80 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 --2- 172.21.15.138:0/2829919590 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcaf403cf00 0x7fcaf403f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7fcaf80e92c0 con 0x7fcb0809ef80 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 client.84668.objecter ms_dispatch 0x7fcb08000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 3 client.84668.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:39.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 3 client.84668.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 client.84668.objecter checking command 1 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 --2- 172.21.15.138:0/2829919590 >> v2:172.21.15.138:6804/81047329 conn(0x7fcaf40443e0 0x7fcaf4046880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 20 client.84668.objecter _get_session s=0x7fcaf4043d30 osd=2 3 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 20 client.84668.objecter put_session s=0x7fcaf4043d30 osd=2 3 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 15 client.84668.objecter _session_command_op_remove -1 1 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 20 client.84668.objecter _get_session s=0x7fcaf4043d30 osd=2 3 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 20 client.84668.objecter get_session s=0x7fcaf4043d30 osd=2 3 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 15 client.84668.objecter _session_command_op_assign 2 1 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 20 client.84668.objecter put_session s=0x7fcaf4043d30 osd=2 4 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 client.84668.objecter _send_command 1 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 --> v2:172.21.15.138:6804/81047329 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcaf80ab600 con 0x7fcaf40443e0 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 20 client.84668.objecter dump_active .. 0 homeless 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=698) ==== 24+0+0 (secure 0 0 0) 0x7fcaf80f0050 con 0x7fcb0809ef80 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb04ff9640 10 monclient: handle_get_version_reply finishing 1 version 698 2024-09-11T05:26:39.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0cd07640 1 --2- 172.21.15.138:0/2829919590 >> v2:172.21.15.138:6804/81047329 conn(0x7fcaf40443e0 0x7fcaf4046880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb0cd07640 10 monclient: get_auth_request con 0x7fcaf40443e0 auth_method 0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb077fe640 1 --2- 172.21.15.138:0/2829919590 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcaf403cf00 0x7fcaf403f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.659+0000 7fcb077fe640 10 monclient: get_auth_request con 0x7fcaf403cf00 auth_method 0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb04ff9640 10 client.84668.objecter ms_handle_connect 0x7fcaf403cf00 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb077fe640 1 --2- 172.21.15.138:0/2829919590 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcaf403cf00 0x7fcaf403f3c0 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fcb081ce180 tx=0x7fcafc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb0cd07640 1 --2- 172.21.15.138:0/2829919590 >> v2:172.21.15.138:6804/81047329 conn(0x7fcaf40443e0 0x7fcaf4046880 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb04ff9640 10 client.84668.objecter ms_handle_connect 0x7fcaf40443e0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 <== osd.2 v2:172.21.15.138:6804/81047329 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcaf80ab600 con 0x7fcaf40443e0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb04ff9640 10 client.84668.objecter ms_dispatch 0x7fcb08000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb04ff9640 10 client.84668.objecter _finish_command 1 = system:0 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb04ff9640 20 client.84668.objecter put_session s=0x7fcaf4043d30 osd=2 4 2024-09-11T05:26:39.670 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.663+0000 7fcb04ff9640 15 client.84668.objecter _session_command_op_remove 2 1 2024-09-11T05:26:39.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 10 client.84668.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T05:26:39.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 15 client.84668.objecter _session_command_op_assign -1 2 2024-09-11T05:26:39.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 20 client.84668.objecter _get_session s=0x7fcaf4043d30 osd=2 3 2024-09-11T05:26:39.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 20 client.84668.objecter put_session s=0x7fcaf4043d30 osd=2 3 2024-09-11T05:26:39.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 20 client.84668.objecter _get_session s=0x7fcaf4043d30 osd=2 3 2024-09-11T05:26:39.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 15 client.84668.objecter _session_command_op_remove -1 2 2024-09-11T05:26:39.675 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 20 client.84668.objecter get_session s=0x7fcaf4043d30 osd=2 3 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 15 client.84668.objecter _session_command_op_assign 2 2 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 20 client.84668.objecter put_session s=0x7fcaf4043d30 osd=2 4 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 10 client.84668.objecter _send_command 2 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 --> v2:172.21.15.138:6804/81047329 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcb081344e0 con 0x7fcaf40443e0 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb04ff9640 1 -- 172.21.15.138:0/2829919590 <== osd.2 v2:172.21.15.138:6804/81047329 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcb081344e0 con 0x7fcaf40443e0 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb04ff9640 10 client.84668.objecter ms_dispatch 0x7fcb08000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb04ff9640 10 client.84668.objecter _finish_command 2 = system:0 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb04ff9640 20 client.84668.objecter put_session s=0x7fcaf4043d30 osd=2 4 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb04ff9640 15 client.84668.objecter _session_command_op_remove 2 2 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:26:39.677 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 10 librados: watch_flush enter 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 10 librados: watch_flush exit 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 10 client.84668.objecter close_session for osd.2 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 >> v2:172.21.15.138:6804/81047329 conn(0x7fcaf40443e0 msgr2=0x7fcaf4046880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 --2- 172.21.15.138:0/2829919590 >> v2:172.21.15.138:6804/81047329 conn(0x7fcaf40443e0 0x7fcaf4046880 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 20 client.84668.objecter put_session s=0x7fcaf4043d30 osd=2 1 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 20 client.84668.objecter shutdown clearing up homeless session... 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 10 client.84668.objecter successfully canceled tick 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcaf403cf00 msgr2=0x7fcaf403f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 --2- 172.21.15.138:0/2829919590 >> v2:172.21.15.190:6800/1562782189 conn(0x7fcaf403cf00 0x7fcaf403f3c0 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fcb081ce180 tx=0x7fcafc008040 comp rx=0 tx=0).stop 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 10 monclient: shutdown 2024-09-11T05:26:39.678 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:39.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809ef80 msgr2=0x7fcb0806fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 --2- 172.21.15.138:0/2829919590 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7fcb0809ef80 0x7fcb0806fb30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcaf801c250 tx=0x7fcaf80191c0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0cd07640 1 -- 172.21.15.138:0/2829919590 reap_dead start 2024-09-11T05:26:39.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 shutdown_connections 2024-09-11T05:26:39.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 >> 172.21.15.138:0/2829919590 conn(0x7fcb080a7870 msgr2=0x7fcb080a1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:39.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 shutdown_connections 2024-09-11T05:26:39.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 -- 172.21.15.138:0/2829919590 wait complete. 2024-09-11T05:26:39.679 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.679+0000 7fcb0e76a640 1 librados: shutdown 2024-09-11T05:26:39.691 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T05:26:39.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient: get_monmap_and_config 2024-09-11T05:26:39.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient: build_initial_monmap 2024-09-11T05:26:39.803 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient: monmap: 2024-09-11T05:26:39.803 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:39.810246+0000 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:39.810246+0000 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:39.804 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 1 Processor -- start 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 1 -- start start 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient: init 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3bd8063d50 0x7f3bd80a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd8063d50 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bd806ef80 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3bd813dc00 0x7f3bd813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd813dc00 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bd80704a0 con 0x7f3bd806ef80 2024-09-11T05:26:39.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f3bd806fd10 con 0x7f3bd813dc00 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3bd8064130 con 0x7f3bd8063d50 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bde1b8640 10 monclient(hunting): authenticate will time out at 2083.246336s 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3bd813dc00 0x7f3bd813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f3bd813dc00 0x7f3bd813fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59694/0 (socket says 172.21.15.138:59694) 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 1 -- 172.21.15.138:0/1143879938 learned_addr learned my addr 172.21.15.138:0/1143879938 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient(hunting): get_auth_request con 0x7f3bd813dc00 auth_method 0 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd77fe640 1 --2- 172.21.15.138:0/1143879938 >> v2:172.21.15.190:3300/0 conn(0x7f3bd8063d50 0x7f3bd80a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:39.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd77fe640 10 monclient(hunting): get_auth_request con 0x7f3bd8063d50 auth_method 0 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient(hunting): handle_auth_done global_id 114369 payload 274 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 1 -- 172.21.15.138:0/1143879938 >> v2:172.21.15.190:3300/0 conn(0x7f3bd8063d50 msgr2=0x7f3bd80a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 1 --2- 172.21.15.138:0/1143879938 >> v2:172.21.15.190:3300/0 conn(0x7f3bd8063d50 0x7f3bd80a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 1 -- 172.21.15.138:0/1143879938 >> v1:172.21.15.190:6789/0 conn(0x7f3bd806ef80 legacy=0x7f3bd80a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient: _finish_hunting 0 2024-09-11T05:26:39.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 1 monclient: found mon.noname-a 2024-09-11T05:26:39.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:39.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:39.810 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.807+0000 7f3bd7fff640 1 -- 172.21.15.138:0/1143879938 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd806f400 con 0x7f3bd813dc00 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 1 -- 172.21.15.138:0/1143879938 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd813dc00 msgr2=0x7f3bd813fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 1 --2- 172.21.15.138:0/1143879938 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd813dc00 0x7f3bd813fff0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80a8f60 tx=0x7f3bd001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47df608314f63f30 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 1 -- 172.21.15.138:0/1143879938 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd00a55c0 con 0x7f3bd813dc00 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: dump: 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:39.811 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: _finish_auth 0 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: _check_auth_tickets 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 1 -- 172.21.15.138:0/1143879938 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bd00a5760 con 0x7f3bd813dc00 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 1 -- 172.21.15.138:0/1143879938 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd00a5b30 con 0x7f3bd813dc00 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 5 monclient: authenticate success, global_id 114369 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 4 set_mon_vals no callback set 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd67fc640 10 monclient: dump: 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:39.812 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.813 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.813 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.813 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:39.813 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.813 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:39.813 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:39.813 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:39.813 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient: shutdown 2024-09-11T05:26:39.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1143879938 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd813dc00 msgr2=0x7f3bd813fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1143879938 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd813dc00 0x7f3bd813fff0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80a8f60 tx=0x7f3bd001c940 comp rx=0 tx=0).stop 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1143879938 shutdown_connections 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1143879938 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd813dc00 0x7f3bd813fff0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1143879938 >> v2:172.21.15.190:3300/0 conn(0x7f3bd8063d50 0x7f3bd80a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1143879938 >> 172.21.15.138:0/1143879938 conn(0x7f3bd80a6180 msgr2=0x7f3bd8063640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1143879938 shutdown_connections 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1143879938 wait complete. 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient: build_initial_monmap 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient: monmap: 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:39.818 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 librados: starting msgr at 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 librados: starting objecter 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 Processor -- start 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- start start 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 librados: setting wanted keys 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 librados: calling monclient init 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient: init 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:26:39.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd8063d50 0x7f3bd8124050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd8063d50 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bd806ef80 0x7f3bd8126110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd806ef80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3bd813dc00 0x7f3bd813cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd813dc00 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd8142f60 con 0x7f3bd8063d50 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd80a5890 con 0x7f3bd806ef80 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd81430e0 con 0x7f3bd813dc00 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bde1b8640 10 monclient(hunting): authenticate will time out at 2083.254556s 2024-09-11T05:26:39.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd8063d50 0x7f3bd8124050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd8063d50 0x7f3bd8124050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59696/0 (socket says 172.21.15.138:59696) 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 -- 172.21.15.138:0/1178692390 learned_addr learned my addr 172.21.15.138:0/1178692390 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient(hunting): get_auth_request con 0x7f3bd8063d50 auth_method 0 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3bd813dc00 0x7f3bd813cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 10 monclient(hunting): get_auth_request con 0x7f3bd813dc00 auth_method 0 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:39.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:39.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:39.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:39.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:39.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd6ffd640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bd806ef80 0x7f3bd8126110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient(hunting): handle_auth_done global_id 114372 payload 824 2024-09-11T05:26:39.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 -- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3bd813dc00 msgr2=0x7f3bd813cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3bd813dc00 0x7f3bd813cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 -- 172.21.15.138:0/1178692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bd806ef80 msgr2=0x7f3bd8126110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bd806ef80 0x7f3bd8126110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient: _finish_hunting 0 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 monclient: found mon.noname-a 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd77fe640 1 -- 172.21.15.138:0/1178692390 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd8127070 con 0x7f3bd8063d50 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd6ffd640 10 monclient: get_auth_request con 0x7f3bd806ef80 auth_method 0 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd6ffd640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bd806ef80 0x7f3bd8126110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.815+0000 7f3bd7fff640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3bd813dc00 0x7f3bd813cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bd77fe640 1 -- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd8063d50 msgr2=0x7f3bd8124050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bd77fe640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd8063d50 0x7f3bd8124050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000e870 tx=0x7f3bc000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 client.?.objecter ms_handle_connect 0x7f3bd8063d50 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 1 -- 172.21.15.138:0/1178692390 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0017070 con 0x7f3bd8063d50 2024-09-11T05:26:39.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: dump: 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:39.825 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: _finish_auth 0 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: _check_auth_tickets 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 5 monclient: authenticate success, global_id 114372 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 1 -- 172.21.15.138:0/1178692390 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc000ee60 con 0x7f3bd8063d50 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 10 monclient: _renew_subs 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd80a8c50 con 0x7f3bd8063d50 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 1 -- 172.21.15.138:0/1178692390 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc000cd30 con 0x7f3bd8063d50 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:39.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 monclient: dump: 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bd5ffb640 4 set_mon_vals no callback set 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 10 client.114372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 10 monclient: _renew_subs 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd81255b0 con 0x7f3bd8063d50 2024-09-11T05:26:39.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 1 librados: init done 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd80a4a30 con 0x7f3bd8063d50 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 1 -- 172.21.15.138:0/1178692390 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f3bc0024060 con 0x7f3bd8063d50 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 1 --2- 172.21.15.138:0/1178692390 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3bb803cef0 0x7f3bb803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 1 -- 172.21.15.138:0/1178692390 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f3bc004f550 con 0x7f3bd8063d50 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 10 client.114372.objecter ms_dispatch 0x7f3bd8000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 3 client.114372.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 3 client.114372.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bcffff640 20 client.114372.objecter dump_active .. 0 homeless 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bd6ffd640 1 --2- 172.21.15.138:0/1178692390 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3bb803cef0 0x7f3bb803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:39.828 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.819+0000 7f3bd6ffd640 10 monclient: get_auth_request con 0x7f3bb803cef0 auth_method 0 2024-09-11T05:26:39.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.823+0000 7f3bcffff640 10 client.114372.objecter ms_handle_connect 0x7f3bb803cef0 2024-09-11T05:26:39.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.823+0000 7f3bd6ffd640 1 --2- 172.21.15.138:0/1178692390 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3bb803cef0 0x7f3bb803f3b0 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8127840 tx=0x7f3bc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:39.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.823+0000 7f3bcffff640 1 -- 172.21.15.138:0/1178692390 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3bd8127070 con 0x7f3bd8063d50 2024-09-11T05:26:39.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.823+0000 7f3bcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:39.829 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.823+0000 7f3bcffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:39.989 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.991+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f3bd8127a30 con 0x7f3bb803cef0 2024-09-11T05:26:39.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.991+0000 7f3bcffff640 1 -- 172.21.15.138:0/1178692390 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3bc80053e0 con 0x7f3bb803cef0 2024-09-11T05:26:39.990 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T05:26:39.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 10 librados: watch_flush enter 2024-09-11T05:26:39.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 10 librados: watch_flush exit 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 20 client.114372.objecter shutdown clearing up homeless session... 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 10 client.114372.objecter successfully canceled tick 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3bb803cef0 msgr2=0x7f3bb803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1178692390 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3bb803cef0 0x7f3bb803f3b0 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8127840 tx=0x7f3bc8009040 comp rx=0 tx=0).stop 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 10 monclient: shutdown 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd8063d50 msgr2=0x7f3bd8124050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd8063d50 0x7f3bd8124050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000e870 tx=0x7f3bc000ed30 comp rx=0 tx=0).stop 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 shutdown_connections 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1178692390 >> v2:172.21.15.190:6800/1562782189 conn(0x7f3bb803cef0 0x7f3bb803f3b0 unknown :-1 s=CLOSED pgs=3528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f3bd813dc00 0x7f3bd813cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bd806ef80 0x7f3bd8126110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 --2- 172.21.15.138:0/1178692390 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f3bd8063d50 0x7f3bd8124050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:39.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 >> 172.21.15.138:0/1178692390 conn(0x7f3bd80a6180 msgr2=0x7f3bd80a8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:39.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 shutdown_connections 2024-09-11T05:26:39.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 -- 172.21.15.138:0/1178692390 wait complete. 2024-09-11T05:26:39.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:39.995+0000 7f3bde1b8640 1 librados: shutdown 2024-09-11T05:26:40.007 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T05:26:40.007 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient: get_monmap_and_config 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient: build_initial_monmap 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient: monmap: 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:40.131430+0000 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:40.131430+0000 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:40.125 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.126 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:40.126 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:40.126 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:40.126 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:40.126 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:40.126 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 1 Processor -- start 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 1 -- start start 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient: init 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9d280a5b40 0x7f9d280a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient(hunting): picked mon.noname-c con 0x7f9d280a5b40 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f9d280a6460 0x7f9d28138900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.129 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient(hunting): picked mon.noname-a con 0x7f9d280a6460 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d28138e40 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d2809cb40 con 0x7f9d28138e40 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f9d2809dd50 con 0x7f9d280a6460 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9d2809d190 con 0x7f9d280a5b40 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.127+0000 7f9d2fc18640 10 monclient(hunting): authenticate will time out at 2083.567950s 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f9d280a6460 0x7f9d28138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f9d280a6460 0x7f9d28138900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59702/0 (socket says 172.21.15.138:59702) 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 1 -- 172.21.15.138:0/1828473838 learned_addr learned my addr 172.21.15.138:0/1828473838 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient(hunting): get_auth_request con 0x7f9d280a6460 auth_method 0 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.130 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d9b4640 1 --2- 172.21.15.138:0/1828473838 >> v2:172.21.15.190:3300/0 conn(0x7f9d280a5b40 0x7f9d280a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d9b4640 10 monclient(hunting): get_auth_request con 0x7f9d280a5b40 auth_method 0 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d9b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d9b4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d9b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient(hunting): handle_auth_done global_id 114375 payload 274 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 1 -- 172.21.15.138:0/1828473838 >> v2:172.21.15.190:3300/0 conn(0x7f9d280a5b40 msgr2=0x7f9d280a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 1 --2- 172.21.15.138:0/1828473838 >> v2:172.21.15.190:3300/0 conn(0x7f9d280a5b40 0x7f9d280a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 1 -- 172.21.15.138:0/1828473838 >> v1:172.21.15.190:6789/0 conn(0x7f9d28138e40 legacy=0x7f9d2813fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.131 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient: _finish_hunting 0 2024-09-11T05:26:40.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 1 monclient: found mon.noname-a 2024-09-11T05:26:40.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:40.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.132 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.131+0000 7f9d2d1b3640 1 -- 172.21.15.138:0/1828473838 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2809ce20 con 0x7f9d280a6460 2024-09-11T05:26:40.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d1b3640 1 -- 172.21.15.138:0/1828473838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d280a6460 msgr2=0x7f9d28138900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:40.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d1b3640 1 --2- 172.21.15.138:0/1828473838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d280a6460 0x7f9d28138900 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d2807cd80 tx=0x7f9d1801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182880f535b341e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 1 -- 172.21.15.138:0/1828473838 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d180b0070 con 0x7f9d280a6460 2024-09-11T05:26:40.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.135 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: dump: 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.136 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: _finish_auth 0 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: _check_auth_tickets 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 5 monclient: authenticate success, global_id 114375 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 1 -- 172.21.15.138:0/1828473838 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d180ab600 con 0x7f9d280a6460 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 4 set_mon_vals no callback set 2024-09-11T05:26:40.137 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 1 -- 172.21.15.138:0/1828473838 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d180ab9d0 con 0x7f9d280a6460 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2c9b2640 10 monclient: dump: 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.138 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient: shutdown 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- 172.21.15.138:0/1828473838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d280a6460 msgr2=0x7f9d28138900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 --2- 172.21.15.138:0/1828473838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d280a6460 0x7f9d28138900 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d2807cd80 tx=0x7f9d1801c470 comp rx=0 tx=0).stop 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- 172.21.15.138:0/1828473838 shutdown_connections 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 --2- 172.21.15.138:0/1828473838 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d280a6460 0x7f9d28138900 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 --2- 172.21.15.138:0/1828473838 >> v2:172.21.15.190:3300/0 conn(0x7f9d280a5b40 0x7f9d280a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- 172.21.15.138:0/1828473838 >> 172.21.15.138:0/1828473838 conn(0x7f9d280ab9f0 msgr2=0x7f9d280a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- 172.21.15.138:0/1828473838 shutdown_connections 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- 172.21.15.138:0/1828473838 wait complete. 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient: build_initial_monmap 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient: monmap: 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:40.139 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 librados: starting msgr at 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 librados: starting objecter 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 Processor -- start 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- start start 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 librados: setting wanted keys 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 librados: calling monclient init 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient: init 2024-09-11T05:26:40.140 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9d280a5b40 0x7f9d281d0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient(hunting): picked mon.noname-c con 0x7f9d280a5b40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d280a6460 0x7f9d28140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient(hunting): picked mon.noname-b con 0x7f9d280a6460 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d28138e40 0x7f9d281405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient(hunting): picked mon.noname-a con 0x7f9d28138e40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f9d280a7380 con 0x7f9d28138e40 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9d280a7200 con 0x7f9d280a6460 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f9d280a6e10 con 0x7f9d280a5b40 2024-09-11T05:26:40.141 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2fc18640 10 monclient(hunting): authenticate will time out at 2083.575306s 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d28138e40 0x7f9d281405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d9b4640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9d280a5b40 0x7f9d281d0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d28138e40 0x7f9d281405a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59716/0 (socket says 172.21.15.138:59716) 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 -- 172.21.15.138:0/2047726713 learned_addr learned my addr 172.21.15.138:0/2047726713 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient(hunting): get_auth_request con 0x7f9d28138e40 auth_method 0 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d9b4640 10 monclient(hunting): get_auth_request con 0x7f9d280a5b40 auth_method 0 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d9b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d9b4640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d9b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.142 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d9b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d9b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d9b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient(hunting): handle_auth_done global_id 114378 payload 824 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2d1b3640 1 --2- 172.21.15.138:0/2047726713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d280a6460 0x7f9d28140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 -- 172.21.15.138:0/2047726713 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9d280a5b40 msgr2=0x7f9d281d0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 --2- 172.21.15.138:0/2047726713 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f9d280a5b40 0x7f9d281d0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 -- 172.21.15.138:0/2047726713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d280a6460 msgr2=0x7f9d28140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 --2- 172.21.15.138:0/2047726713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d280a6460 0x7f9d28140060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient: _finish_hunting 0 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 monclient: found mon.noname-a 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.143 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.135+0000 7f9d2e1b5640 1 -- 172.21.15.138:0/2047726713 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d280a1e60 con 0x7f9d28138e40 2024-09-11T05:26:40.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2e1b5640 1 -- 172.21.15.138:0/2047726713 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d28138e40 msgr2=0x7f9d281405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:40.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2e1b5640 1 --2- 172.21.15.138:0/2047726713 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d28138e40 0x7f9d281405a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000c780 tx=0x7f9d2000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 client.?.objecter ms_handle_connect 0x7f9d28138e40 2024-09-11T05:26:40.146 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2000ac00 con 0x7f9d28138e40 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: dump: 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.147 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: _finish_auth 0 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 5 monclient: authenticate success, global_id 114378 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d2000ada0 con 0x7f9d28138e40 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 10 monclient: _renew_subs 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d28140c90 con 0x7f9d28138e40 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d20004bb0 con 0x7f9d28138e40 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: dump: 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.148 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1ffff640 4 set_mon_vals no callback set 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 10 client.114378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 10 monclient: _renew_subs 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d2813bb00 con 0x7f9d28138e40 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 1 librados: init done 2024-09-11T05:26:40.149 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 10 client.114378.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 15 client.114378.objecter _session_command_op_assign -1 1 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 20 client.114378.objecter _get_session osd=-1 returning homeless 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 10 client.114378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9d280a5f20 con 0x7f9d28138e40 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f9d20023060 con 0x7f9d28138e40 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 --2- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9d0c03ced0 0x7f9d0c03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f9d2004f950 con 0x7f9d28138e40 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 client.114378.objecter ms_dispatch 0x7f9d28000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 3 client.114378.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 3 client.114378.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 client.114378.objecter checking command 1 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 --2- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6809/3046884208 conn(0x7f9d0c0443b0 0x7f9d0c046850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 20 client.114378.objecter _get_session s=0x7f9d0c043d00 osd=3 3 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 20 client.114378.objecter put_session s=0x7f9d0c043d00 osd=3 3 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 15 client.114378.objecter _session_command_op_remove -1 1 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 20 client.114378.objecter _get_session s=0x7f9d0c043d00 osd=3 3 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 20 client.114378.objecter get_session s=0x7f9d0c043d00 osd=3 3 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 15 client.114378.objecter _session_command_op_assign 3 1 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 20 client.114378.objecter put_session s=0x7f9d0c043d00 osd=3 4 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 client.114378.objecter _send_command 1 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 --> v2:172.21.15.190:6809/3046884208 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d20009eb0 con 0x7f9d0c0443b0 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 20 client.114378.objecter dump_active .. 0 homeless 2024-09-11T05:26:40.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_get_version_reply(handle=1 version=698) ==== 24+0+0 (secure 0 0 0) 0x7f9d200542a0 con 0x7f9d28138e40 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d1e7fc640 10 monclient: handle_get_version_reply finishing 1 version 698 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2d9b4640 1 --2- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9d0c03ced0 0x7f9d0c03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2d9b4640 10 monclient: get_auth_request con 0x7f9d0c03ced0 auth_method 0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2d1b3640 1 --2- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6809/3046884208 conn(0x7f9d0c0443b0 0x7f9d0c046850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.139+0000 7f9d2d1b3640 10 monclient: get_auth_request con 0x7f9d0c0443b0 auth_method 0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.143+0000 7f9d1e7fc640 10 client.114378.objecter ms_handle_connect 0x7f9d0c03ced0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.143+0000 7f9d2d9b4640 1 --2- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9d0c03ced0 0x7f9d0c03f390 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7f9d281d1c80 tx=0x7f9d24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.147+0000 7f9d1e7fc640 10 client.114378.objecter ms_handle_connect 0x7f9d0c0443b0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.147+0000 7f9d2d1b3640 1 --2- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6809/3046884208 conn(0x7f9d0c0443b0 0x7f9d0c046850 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.147+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 <== osd.3 v2:172.21.15.190:6809/3046884208 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d20009eb0 con 0x7f9d0c0443b0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.147+0000 7f9d1e7fc640 10 client.114378.objecter ms_dispatch 0x7f9d28000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.147+0000 7f9d1e7fc640 10 client.114378.objecter _finish_command 1 = system:0 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.147+0000 7f9d1e7fc640 20 client.114378.objecter put_session s=0x7f9d0c043d00 osd=3 4 2024-09-11T05:26:40.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.147+0000 7f9d1e7fc640 15 client.114378.objecter _session_command_op_remove 3 1 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 10 client.114378.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 15 client.114378.objecter _session_command_op_assign -1 2 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 20 client.114378.objecter _get_session s=0x7f9d0c043d00 osd=3 3 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 20 client.114378.objecter put_session s=0x7f9d0c043d00 osd=3 3 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 20 client.114378.objecter _get_session s=0x7f9d0c043d00 osd=3 3 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 15 client.114378.objecter _session_command_op_remove -1 2 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 20 client.114378.objecter get_session s=0x7f9d0c043d00 osd=3 3 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 15 client.114378.objecter _session_command_op_assign 3 2 2024-09-11T05:26:40.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 20 client.114378.objecter put_session s=0x7f9d0c043d00 osd=3 4 2024-09-11T05:26:40.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 10 client.114378.objecter _send_command 2 2024-09-11T05:26:40.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 --> v2:172.21.15.190:6809/3046884208 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9d280a8b10 con 0x7f9d0c0443b0 2024-09-11T05:26:40.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d1e7fc640 1 -- 172.21.15.138:0/2047726713 <== osd.3 v2:172.21.15.190:6809/3046884208 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9d280a8b10 con 0x7f9d0c0443b0 2024-09-11T05:26:40.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d1e7fc640 10 client.114378.objecter ms_dispatch 0x7f9d28000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:26:40.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d1e7fc640 10 client.114378.objecter _finish_command 2 = system:0 2024-09-11T05:26:40.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d1e7fc640 20 client.114378.objecter put_session s=0x7f9d0c043d00 osd=3 4 2024-09-11T05:26:40.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.159+0000 7f9d1e7fc640 15 client.114378.objecter _session_command_op_remove 3 2 2024-09-11T05:26:40.159 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:40.159 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:26:40.159 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:40.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 10 librados: watch_flush enter 2024-09-11T05:26:40.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 10 librados: watch_flush exit 2024-09-11T05:26:40.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 10 client.114378.objecter close_session for osd.3 2024-09-11T05:26:40.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6809/3046884208 conn(0x7f9d0c0443b0 msgr2=0x7f9d0c046850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 --2- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6809/3046884208 conn(0x7f9d0c0443b0 0x7f9d0c046850 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 20 client.114378.objecter put_session s=0x7f9d0c043d00 osd=3 1 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 20 client.114378.objecter shutdown clearing up homeless session... 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 10 client.114378.objecter successfully canceled tick 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9d0c03ced0 msgr2=0x7f9d0c03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 --2- 172.21.15.138:0/2047726713 >> v2:172.21.15.190:6800/1562782189 conn(0x7f9d0c03ced0 0x7f9d0c03f390 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7f9d281d1c80 tx=0x7f9d24008040 comp rx=0 tx=0).stop 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 10 monclient: shutdown 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d28138e40 msgr2=0x7f9d281405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 --2- 172.21.15.138:0/2047726713 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f9d28138e40 0x7f9d281405a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000c780 tx=0x7f9d2000cc40 comp rx=0 tx=0).stop 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2e1b5640 1 -- 172.21.15.138:0/2047726713 reap_dead start 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 shutdown_connections 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 >> 172.21.15.138:0/2047726713 conn(0x7f9d280ab9f0 msgr2=0x7f9d280a32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 shutdown_connections 2024-09-11T05:26:40.161 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 -- 172.21.15.138:0/2047726713 wait complete. 2024-09-11T05:26:40.162 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.163+0000 7f9d2fc18640 1 librados: shutdown 2024-09-11T05:26:40.174 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T05:26:40.286 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient: get_monmap_and_config 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient: build_initial_monmap 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient: monmap: 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:40.293484+0000 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:40.293484+0000 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:40.287 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:40.288 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:40.288 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:40.288 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 1 Processor -- start 2024-09-11T05:26:40.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 1 -- start start 2024-09-11T05:26:40.289 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient: init 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7efcbc070010 0x7efcbc135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc070010 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efcbc09e6e0 0x7efcbc1382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc09e6e0 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efcbc09eb10 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcbc0a8990 con 0x7efcbc09eb10 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7efcbc0703f0 con 0x7efcbc070010 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efcbc09cbf0 con 0x7efcbc09e6e0 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcc20fd640 10 monclient(hunting): authenticate will time out at 2083.729562s 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7efcbc070010 0x7efcbc135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.290 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7efcbc070010 0x7efcbc135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59724/0 (socket says 172.21.15.138:59724) 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 1 -- 172.21.15.138:0/3860276976 learned_addr learned my addr 172.21.15.138:0/3860276976 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient(hunting): get_auth_request con 0x7efcbc070010 auth_method 0 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbaffd640 1 --2- 172.21.15.138:0/3860276976 >> v2:172.21.15.190:3300/0 conn(0x7efcbc09e6e0 0x7efcbc1382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient(hunting): handle_auth_done global_id 114381 payload 274 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 1 -- 172.21.15.138:0/3860276976 >> v2:172.21.15.190:3300/0 conn(0x7efcbc09e6e0 msgr2=0x7efcbc1382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 1 --2- 172.21.15.138:0/3860276976 >> v2:172.21.15.190:3300/0 conn(0x7efcbc09e6e0 0x7efcbc1382c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.291 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 1 -- 172.21.15.138:0/3860276976 >> v1:172.21.15.190:6789/0 conn(0x7efcbc09eb10 legacy=0x7efcbc138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:26:40.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 1 monclient: found mon.noname-a 2024-09-11T05:26:40.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:40.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.292 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.291+0000 7efcbb7fe640 1 -- 172.21.15.138:0/3860276976 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc09ced0 con 0x7efcbc070010 2024-09-11T05:26:40.293 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcbb7fe640 1 -- 172.21.15.138:0/3860276976 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc070010 msgr2=0x7efcbc135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcbb7fe640 1 --2- 172.21.15.138:0/3860276976 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc070010 0x7efcbc135450 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efcbc06efb0 tx=0x7efcb401c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff8441cc89ff06c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 1 -- 172.21.15.138:0/3860276976 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb40a5720 con 0x7efcbc070010 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: dump: 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.294 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: _finish_auth 0 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: _check_auth_tickets 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 1 -- 172.21.15.138:0/3860276976 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb40a58c0 con 0x7efcbc070010 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 1 -- 172.21.15.138:0/3860276976 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb40a5c90 con 0x7efcbc070010 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 5 monclient: authenticate success, global_id 114381 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 4 set_mon_vals no callback set 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:40.295 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcba7fc640 10 monclient: dump: 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 10 monclient: shutdown 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:40.296 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 -- 172.21.15.138:0/3860276976 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc070010 msgr2=0x7efcbc135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 --2- 172.21.15.138:0/3860276976 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc070010 0x7efcbc135450 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efcbc06efb0 tx=0x7efcb401c880 comp rx=0 tx=0).stop 2024-09-11T05:26:40.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 -- 172.21.15.138:0/3860276976 shutdown_connections 2024-09-11T05:26:40.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 --2- 172.21.15.138:0/3860276976 >> v2:172.21.15.190:3300/0 conn(0x7efcbc09e6e0 0x7efcbc1382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 --2- 172.21.15.138:0/3860276976 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc070010 0x7efcbc135450 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 -- 172.21.15.138:0/3860276976 >> 172.21.15.138:0/3860276976 conn(0x7efcbc0ab830 msgr2=0x7efcbc0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:40.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 -- 172.21.15.138:0/3860276976 shutdown_connections 2024-09-11T05:26:40.297 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 -- 172.21.15.138:0/3860276976 wait complete. 2024-09-11T05:26:40.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 10 monclient: build_initial_monmap 2024-09-11T05:26:40.299 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 10 monclient: monmap: 2024-09-11T05:26:40.299 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 librados: starting msgr at 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.295+0000 7efcc20fd640 1 librados: starting objecter 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 Processor -- start 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 -- start start 2024-09-11T05:26:40.300 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 librados: setting wanted keys 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 librados: calling monclient init 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient: init 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc070010 0x7efcbc1d0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient(hunting): picked mon.noname-b con 0x7efcbc070010 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc09e6e0 0x7efcbc1d0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc09e6e0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcbc09eb10 0x7efcbc1d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc09eb10 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc142e80 con 0x7efcbc09e6e0 2024-09-11T05:26:40.301 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc139120 con 0x7efcbc070010 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc143000 con 0x7efcbc09eb10 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcc20fd640 10 monclient(hunting): authenticate will time out at 2083.736651s 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc09e6e0 0x7efcbc1d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc09e6e0 0x7efcbc1d0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59732/0 (socket says 172.21.15.138:59732) 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 -- 172.21.15.138:0/4178428916 learned_addr learned my addr 172.21.15.138:0/4178428916 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient(hunting): get_auth_request con 0x7efcbc09e6e0 auth_method 0 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbbfff640 1 --2- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcbc09eb10 0x7efcbc1d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbbfff640 10 monclient(hunting): get_auth_request con 0x7efcbc09eb10 auth_method 0 2024-09-11T05:26:40.302 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient(hunting): handle_auth_done global_id 114384 payload 824 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 -- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcbc09eb10 msgr2=0x7efcbc1d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 --2- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcbc09eb10 0x7efcbc1d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 -- 172.21.15.138:0/4178428916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc070010 msgr2=0x7efcbc1d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 --2- 172.21.15.138:0/4178428916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc070010 0x7efcbc1d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient: _finish_hunting 0 2024-09-11T05:26:40.303 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 monclient: found mon.noname-a 2024-09-11T05:26:40.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:40.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 -- 172.21.15.138:0/4178428916 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc0a1d10 con 0x7efcbc09e6e0 2024-09-11T05:26:40.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 -- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc09e6e0 msgr2=0x7efcbc1d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:40.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efcbaffd640 1 --2- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc09e6e0 0x7efcbc1d0940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c7f0 tx=0x7efcb000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.304 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.299+0000 7efca3fff640 1 -- 172.21.15.138:0/4178428916 DispatchQueue::entry inject delay of 0.002000 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 client.?.objecter ms_handle_connect 0x7efcbc09e6e0 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 1 -- 172.21.15.138:0/4178428916 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0015070 con 0x7efcbc09e6e0 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: dump: 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.306 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: _finish_auth 0 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: _check_auth_tickets 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 1 -- 172.21.15.138:0/4178428916 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb0004240 con 0x7efcbc09e6e0 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 1 -- 172.21.15.138:0/4178428916 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0007830 con 0x7efcbc09e6e0 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 5 monclient: authenticate success, global_id 114384 2024-09-11T05:26:40.307 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 monclient: dump: 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 10 monclient: _renew_subs 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcbc0a2c00 con 0x7efcbc09e6e0 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcb9ffb640 4 set_mon_vals no callback set 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 10 client.114384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 10 monclient: _renew_subs 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcbc0a4b80 con 0x7efcbc09e6e0 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 1 librados: init done 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 1 -- 172.21.15.138:0/4178428916 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7efcb00079d0 con 0x7efcbc09e6e0 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcbc135450 con 0x7efcbc09e6e0 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 1 --2- 172.21.15.138:0/4178428916 >> v2:172.21.15.190:6800/1562782189 conn(0x7efca803cef0 0x7efca803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.309 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 1 -- 172.21.15.138:0/4178428916 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7efcb004f6f0 con 0x7efcbc09e6e0 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 10 client.114384.objecter ms_dispatch 0x7efcbc000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 3 client.114384.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 3 client.114384.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efca3fff640 20 client.114384.objecter dump_active .. 0 homeless 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcbb7fe640 1 --2- 172.21.15.138:0/4178428916 >> v2:172.21.15.190:6800/1562782189 conn(0x7efca803cef0 0x7efca803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.303+0000 7efcbb7fe640 10 monclient: get_auth_request con 0x7efca803cef0 auth_method 0 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.307+0000 7efca3fff640 10 client.114384.objecter ms_handle_connect 0x7efca803cef0 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.307+0000 7efcbb7fe640 1 --2- 172.21.15.138:0/4178428916 >> v2:172.21.15.190:6800/1562782189 conn(0x7efca803cef0 0x7efca803f3b0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7efcbc135200 tx=0x7efcb4004ac0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.307+0000 7efca3fff640 1 -- 172.21.15.138:0/4178428916 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7efcb004e130 con 0x7efcbc09e6e0 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.307+0000 7efca3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:40.310 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.307+0000 7efca3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:40.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.475+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7efcbc133680 con 0x7efca803cef0 2024-09-11T05:26:40.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.475+0000 7efca3fff640 1 -- 172.21.15.138:0/4178428916 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7efcbc133680 con 0x7efca803cef0 2024-09-11T05:26:40.474 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T05:26:40.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 10 librados: watch_flush enter 2024-09-11T05:26:40.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 10 librados: watch_flush exit 2024-09-11T05:26:40.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 20 client.114384.objecter shutdown clearing up homeless session... 2024-09-11T05:26:40.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 10 client.114384.objecter successfully canceled tick 2024-09-11T05:26:40.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 >> v2:172.21.15.190:6800/1562782189 conn(0x7efca803cef0 msgr2=0x7efca803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 --2- 172.21.15.138:0/4178428916 >> v2:172.21.15.190:6800/1562782189 conn(0x7efca803cef0 0x7efca803f3b0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7efcbc135200 tx=0x7efcb4004ac0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 10 monclient: shutdown 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc09e6e0 msgr2=0x7efcbc1d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 --2- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc09e6e0 0x7efcbc1d0940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c7f0 tx=0x7efcb000ccb0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 shutdown_connections 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 --2- 172.21.15.138:0/4178428916 >> v2:172.21.15.190:6800/1562782189 conn(0x7efca803cef0 0x7efca803f3b0 unknown :-1 s=CLOSED pgs=3530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 --2- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7efcbc09eb10 0x7efcbc1d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 --2- 172.21.15.138:0/4178428916 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7efcbc09e6e0 0x7efcbc1d0940 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 --2- 172.21.15.138:0/4178428916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcbc070010 0x7efcbc1d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 >> 172.21.15.138:0/4178428916 conn(0x7efcbc0ab830 msgr2=0x7efcbc06ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 shutdown_connections 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 -- 172.21.15.138:0/4178428916 wait complete. 2024-09-11T05:26:40.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.479+0000 7efcc20fd640 1 librados: shutdown 2024-09-11T05:26:40.491 INFO:tasks.ceph:Scrubbing osd.4 2024-09-11T05:26:40.491 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:26:40.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient: get_monmap_and_config 2024-09-11T05:26:40.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient: build_initial_monmap 2024-09-11T05:26:40.611 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient: monmap: 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:40.618275+0000 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:40.618275+0000 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:40.612 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 1 Processor -- start 2024-09-11T05:26:40.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 1 -- start start 2024-09-11T05:26:40.614 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient: init 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f123409da20 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1234074210 0x7f1234074680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient(hunting): picked mon.noname-c con 0x7f1234074210 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient(hunting): picked mon.noname-b con 0x7f123413b410 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12340a2cf0 con 0x7f123409da20 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f123409de00 con 0x7f1234074210 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f12340a2fd0 con 0x7f123413b410 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f123886b640 10 monclient(hunting): authenticate will time out at 2084.054964s 2024-09-11T05:26:40.615 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55104/0 (socket says 172.21.15.138:55104) 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 1 -- 172.21.15.138:0/1374863202 learned_addr learned my addr 172.21.15.138:0/1374863202 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient(hunting): get_auth_request con 0x7f123413b410 auth_method 0 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f122bfff640 1 --2- 172.21.15.138:0/1374863202 >> v2:172.21.15.190:3300/0 conn(0x7f1234074210 0x7f1234074680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f122bfff640 10 monclient(hunting): get_auth_request con 0x7f1234074210 auth_method 0 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f122bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f122bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f122bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient(hunting): handle_auth_done global_id 84680 payload 274 2024-09-11T05:26:40.616 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 1 -- 172.21.15.138:0/1374863202 >> v2:172.21.15.190:3300/0 conn(0x7f1234074210 msgr2=0x7f1234074680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 1 --2- 172.21.15.138:0/1374863202 >> v2:172.21.15.190:3300/0 conn(0x7f1234074210 0x7f1234074680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 1 -- 172.21.15.138:0/1374863202 >> v1:172.21.15.190:6789/0 conn(0x7f123409da20 legacy=0x7f1234132b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient: _finish_hunting 0 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 1 monclient: found mon.noname-b 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.615+0000 7f1232d9d640 1 -- 172.21.15.138:0/1374863202 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12340a32b0 con 0x7f123413b410 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.619+0000 7f1232d9d640 1 -- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 msgr2=0x7f123413d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.619+0000 7f1232d9d640 1 -- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 msgr2=0x7f123413d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T05:26:40.617 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.619+0000 7f1232d9d640 1 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f122c009e50 tx=0x7f122c01c8f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T05:26:40.618 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.619+0000 7f1232d9d640 1 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T05:26:40.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1232d9d640 1 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1232d9d640 10 monclient: get_auth_request con 0x7f123413b410 auth_method 0 2024-09-11T05:26:40.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1232d9d640 0 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1232d9d640 1 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 10 monclient: ms_handle_reset current mon [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 1 -- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 msgr2=0x7f123413d800 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 1 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413b410 0x7f123413d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 10 monclient: _add_conns ranks=[0,3] 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1220003510 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 1 --2- 172.21.15.138:0/1374863202 >> v2:172.21.15.138:3300/0 conn(0x7f12200074a0 0x7f1220009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 10 monclient(hunting): picked mon.noname-a con 0x7f12200074a0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 1 -- 172.21.15.138:0/1374863202 --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122000a270 con 0x7f1220003510 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 1 -- 172.21.15.138:0/1374863202 --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f122000b470 con 0x7f12200074a0 2024-09-11T05:26:40.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f1231d9b640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 1 --2- 172.21.15.138:0/1374863202 >> v2:172.21.15.138:3300/0 conn(0x7f12200074a0 0x7f1220009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient(hunting): get_auth_request con 0x7f12200074a0 auth_method 0 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient(hunting): handle_auth_done global_id 84680 payload 293 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 1 -- 172.21.15.138:0/1374863202 >> v1:172.21.15.138:6789/0 conn(0x7f1220003510 legacy=0x7f1220005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 1 monclient: found mon.noname-a 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:40.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:40.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 1 -- 172.21.15.138:0/1374863202 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122000b730 con 0x7f12200074a0 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 1 -- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12200074a0 msgr2=0x7f1220009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.823+0000 7f122bfff640 1 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12200074a0 0x7f1220009960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f121c00b980 tx=0x7f122c009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2160c8ed2ccc010b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 1 -- 172.21.15.138:0/1374863202 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c00b600 con 0x7f12200074a0 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: dump: 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.824 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: _finish_auth 0 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: _check_auth_tickets 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 1 -- 172.21.15.138:0/1374863202 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121c019070 con 0x7f12200074a0 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 1 -- 172.21.15.138:0/1374863202 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c00dca0 con 0x7f12200074a0 2024-09-11T05:26:40.825 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 5 monclient: authenticate success, global_id 84680 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 4 set_mon_vals no callback set 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1231d9b640 10 monclient: dump: 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.826 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient: shutdown 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12200074a0 msgr2=0x7f1220009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12200074a0 0x7f1220009960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f121c00b980 tx=0x7f122c009810 comp rx=0 tx=0).stop 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f1232d9d640 1 -- 172.21.15.138:0/1374863202 reap_dead start 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- 172.21.15.138:0/1374863202 shutdown_connections 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 --2- 172.21.15.138:0/1374863202 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f12200074a0 0x7f1220009960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- 172.21.15.138:0/1374863202 >> 172.21.15.138:0/1374863202 conn(0x7f12340a7870 msgr2=0x7f12340a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- 172.21.15.138:0/1374863202 shutdown_connections 2024-09-11T05:26:40.827 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- 172.21.15.138:0/1374863202 wait complete. 2024-09-11T05:26:40.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient: build_initial_monmap 2024-09-11T05:26:40.832 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient: monmap: 2024-09-11T05:26:40.832 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:40.832 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 librados: starting msgr at 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 librados: starting objecter 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 Processor -- start 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- start start 2024-09-11T05:26:40.833 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 librados: setting wanted keys 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 librados: calling monclient init 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient: init 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12200074a0 0x7f12341cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient(hunting): picked mon.noname-b con 0x7f12200074a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413a1a0 0x7f123413a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient(hunting): picked mon.noname-c con 0x7f123413a1a0 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1234133c60 0x7f12341340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient(hunting): picked mon.noname-a con 0x7f1234133c60 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f12341408f0 con 0x7f1234133c60 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1234140770 con 0x7f12200074a0 2024-09-11T05:26:40.834 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1234140a70 con 0x7f123413a1a0 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f123886b640 10 monclient(hunting): authenticate will time out at 2084.267845s 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.827+0000 7f122bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413a1a0 0x7f123413a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413a1a0 0x7f123413a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55128/0 (socket says 172.21.15.138:55128) 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 -- 172.21.15.138:0/927858531 learned_addr learned my addr 172.21.15.138:0/927858531 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient(hunting): get_auth_request con 0x7f123413a1a0 auth_method 0 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f1232d9d640 1 --2- 172.21.15.138:0/927858531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1234133c60 0x7f12341340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.835 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123259c640 1 --2- 172.21.15.138:0/927858531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12200074a0 0x7f12341cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f1232d9d640 10 monclient(hunting): get_auth_request con 0x7f1234133c60 auth_method 0 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f1232d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f1232d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f1232d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123259c640 10 monclient(hunting): get_auth_request con 0x7f12200074a0 auth_method 0 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f1232d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f1232d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f1232d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.836 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient(hunting): handle_auth_done global_id 84683 payload 824 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 -- 172.21.15.138:0/927858531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12200074a0 msgr2=0x7f12341cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 --2- 172.21.15.138:0/927858531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f12200074a0 0x7f12341cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 -- 172.21.15.138:0/927858531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1234133c60 msgr2=0x7f12341340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 --2- 172.21.15.138:0/927858531 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f1234133c60 0x7f12341340f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient: _finish_hunting 0 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 monclient: found mon.noname-c 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 -- 172.21.15.138:0/927858531 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1234131510 con 0x7f123413a1a0 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 -- 172.21.15.138:0/927858531 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413a1a0 msgr2=0x7f123413a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122bfff640 1 --2- 172.21.15.138:0/927858531 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413a1a0 0x7f123413a600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f121c008010 tx=0x7f121c004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.837 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 client.?.objecter ms_handle_connect 0x7f123413a1a0 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c004460 con 0x7f123413a1a0 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: dump: 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.838 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: _finish_auth 0 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: _check_auth_tickets 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121c003620 con 0x7f123413a1a0 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 5 monclient: authenticate success, global_id 84683 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 10 monclient: _renew_subs 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123409e1f0 con 0x7f123413a1a0 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c003a10 con 0x7f123413a1a0 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f122affd640 10 monclient: dump: 2024-09-11T05:26:40.839 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123159a640 4 set_mon_vals no callback set 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 10 client.84683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 10 monclient: _renew_subs 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12341356d0 con 0x7f123413a1a0 2024-09-11T05:26:40.840 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.831+0000 7f123886b640 1 librados: init done 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123886b640 10 client.84683.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123886b640 15 client.84683.objecter _session_command_op_assign -1 1 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123886b640 20 client.84683.objecter _get_session osd=-1 returning homeless 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123886b640 10 client.84683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123886b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1234074700 con 0x7f123413a1a0 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f121c013070 con 0x7f123413a1a0 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 1 --2- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6800/1562782189 conn(0x7f121403cea0 0x7f121403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f121c050c50 con 0x7f123413a1a0 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 10 client.84683.objecter ms_dispatch 0x7f1234000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 3 client.84683.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 3 client.84683.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 10 client.84683.objecter checking command 1 2024-09-11T05:26:40.841 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 1 --2- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1214044380 0x7f1214046820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 20 client.84683.objecter _get_session s=0x7f1214043cd0 osd=4 3 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 20 client.84683.objecter put_session s=0x7f1214043cd0 osd=4 3 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 15 client.84683.objecter _session_command_op_remove -1 1 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 20 client.84683.objecter _get_session s=0x7f1214043cd0 osd=4 3 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 20 client.84683.objecter get_session s=0x7f1214043cd0 osd=4 3 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 15 client.84683.objecter _session_command_op_assign 4 1 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 20 client.84683.objecter put_session s=0x7f1214043cd0 osd=4 4 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 10 client.84683.objecter _send_command 1 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 --> v2:172.21.15.190:6801/1118550024 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f121c00ae50 con 0x7f1214044380 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 20 client.84683.objecter dump_active .. 0 homeless 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_get_version_reply(handle=1 version=698) ==== 24+0+0 (secure 0 0 0) 0x7f121c057050 con 0x7f123413a1a0 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 10 monclient: handle_get_version_reply finishing 1 version 698 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123259c640 1 --2- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6800/1562782189 conn(0x7f121403cea0 0x7f121403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.842 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123259c640 10 monclient: get_auth_request con 0x7f121403cea0 auth_method 0 2024-09-11T05:26:40.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f1232d9d640 1 --2- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1214044380 0x7f1214046820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f1232d9d640 10 monclient: get_auth_request con 0x7f1214044380 auth_method 0 2024-09-11T05:26:40.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f122affd640 10 client.84683.objecter ms_handle_connect 0x7f121403cea0 2024-09-11T05:26:40.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.835+0000 7f123259c640 1 --2- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6800/1562782189 conn(0x7f121403cea0 0x7f121403f360 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f1224007fd0 tx=0x7f1224007dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.839+0000 7f122affd640 10 client.84683.objecter ms_handle_connect 0x7f1214044380 2024-09-11T05:26:40.843 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.839+0000 7f1232d9d640 1 --2- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1214044380 0x7f1214046820 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.839+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 <== osd.4 v2:172.21.15.190:6801/1118550024 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f121c00ae50 con 0x7f1214044380 2024-09-11T05:26:40.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.839+0000 7f122affd640 10 client.84683.objecter ms_dispatch 0x7f1234000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:26:40.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.839+0000 7f122affd640 10 client.84683.objecter _finish_command 1 = system:0 2024-09-11T05:26:40.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.839+0000 7f122affd640 20 client.84683.objecter put_session s=0x7f1214043cd0 osd=4 4 2024-09-11T05:26:40.844 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.839+0000 7f122affd640 15 client.84683.objecter _session_command_op_remove 4 1 2024-09-11T05:26:40.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 10 client.84683.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T05:26:40.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 15 client.84683.objecter _session_command_op_assign -1 2 2024-09-11T05:26:40.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 20 client.84683.objecter _get_session s=0x7f1214043cd0 osd=4 3 2024-09-11T05:26:40.851 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 20 client.84683.objecter put_session s=0x7f1214043cd0 osd=4 3 2024-09-11T05:26:40.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 20 client.84683.objecter _get_session s=0x7f1214043cd0 osd=4 3 2024-09-11T05:26:40.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 15 client.84683.objecter _session_command_op_remove -1 2 2024-09-11T05:26:40.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 20 client.84683.objecter get_session s=0x7f1214043cd0 osd=4 3 2024-09-11T05:26:40.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 15 client.84683.objecter _session_command_op_assign 4 2 2024-09-11T05:26:40.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 20 client.84683.objecter put_session s=0x7f1214043cd0 osd=4 4 2024-09-11T05:26:40.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 10 client.84683.objecter _send_command 2 2024-09-11T05:26:40.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 --> v2:172.21.15.190:6801/1118550024 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f123406cd40 con 0x7f1214044380 2024-09-11T05:26:40.852 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f122affd640 1 -- 172.21.15.138:0/927858531 <== osd.4 v2:172.21.15.190:6801/1118550024 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f123406cd40 con 0x7f1214044380 2024-09-11T05:26:40.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f122affd640 10 client.84683.objecter ms_dispatch 0x7f1234000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:26:40.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f122affd640 10 client.84683.objecter _finish_command 2 = system:0 2024-09-11T05:26:40.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f122affd640 20 client.84683.objecter put_session s=0x7f1214043cd0 osd=4 4 2024-09-11T05:26:40.853 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f122affd640 15 client.84683.objecter _session_command_op_remove 4 2 2024-09-11T05:26:40.853 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:40.853 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:26:40.853 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 10 librados: watch_flush enter 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 10 librados: watch_flush exit 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 10 client.84683.objecter close_session for osd.4 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1214044380 msgr2=0x7f1214046820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 --2- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6801/1118550024 conn(0x7f1214044380 0x7f1214046820 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 20 client.84683.objecter put_session s=0x7f1214043cd0 osd=4 1 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 20 client.84683.objecter shutdown clearing up homeless session... 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 10 client.84683.objecter successfully canceled tick 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6800/1562782189 conn(0x7f121403cea0 msgr2=0x7f121403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 --2- 172.21.15.138:0/927858531 >> v2:172.21.15.190:6800/1562782189 conn(0x7f121403cea0 0x7f121403f360 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f1224007fd0 tx=0x7f1224007dc0 comp rx=0 tx=0).stop 2024-09-11T05:26:40.855 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 10 monclient: shutdown 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413a1a0 msgr2=0x7f123413a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 --2- 172.21.15.138:0/927858531 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f123413a1a0 0x7f123413a600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f121c008010 tx=0x7f121c004740 comp rx=0 tx=0).stop 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f1232d9d640 1 -- 172.21.15.138:0/927858531 reap_dead start 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 shutdown_connections 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 >> 172.21.15.138:0/927858531 conn(0x7f12340a7870 msgr2=0x7f12340a5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 shutdown_connections 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 -- 172.21.15.138:0/927858531 wait complete. 2024-09-11T05:26:40.856 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.855+0000 7f123886b640 1 librados: shutdown 2024-09-11T05:26:40.869 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-11T05:26:40.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient: get_monmap_and_config 2024-09-11T05:26:40.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient: build_initial_monmap 2024-09-11T05:26:40.988 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient: monmap: 2024-09-11T05:26:40.988 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:40.988 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.988 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:40.994668+0000 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:40.994668+0000 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:40.989 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.991 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 1 Processor -- start 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 1 -- start start 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient: init 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04100a32c0 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f041009eb30 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f041009ef50 0x7f041013e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient(hunting): picked mon.noname-c con 0x7f041009ef50 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04100a1b50 con 0x7f04100a32c0 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f041009f560 con 0x7f041009eb30 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:40.992 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f041009fe70 con 0x7f041009ef50 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f04168e7640 10 monclient(hunting): authenticate will time out at 2084.431546s 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f040ffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f04100a32c0 0x7f041009e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34784/0 (socket says 172.21.15.138:34784) 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f040ffff640 1 -- 172.21.15.138:0/3890910647 learned_addr learned my addr 172.21.15.138:0/3890910647 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.991+0000 7f0414e84640 1 --2- 172.21.15.138:0/3890910647 >> v2:172.21.15.190:3300/0 conn(0x7f041009ef50 0x7f041013e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient(hunting): get_auth_request con 0x7f041009ef50 auth_method 0 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient(hunting): handle_auth_done global_id 134155 payload 274 2024-09-11T05:26:40.993 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 1 -- 172.21.15.138:0/3890910647 >> v1:172.21.15.190:6789/0 conn(0x7f041009eb30 legacy=0x7f041013a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 1 -- 172.21.15.138:0/3890910647 >> v1:172.21.15.138:6789/0 conn(0x7f04100a32c0 legacy=0x7f041009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:40.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient: _finish_hunting 0 2024-09-11T05:26:40.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 1 monclient: found mon.noname-c 2024-09-11T05:26:40.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:40.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.190:3300/0 2024-09-11T05:26:40.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.995+0000 7f0414e84640 1 -- 172.21.15.138:0/3890910647 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041009f840 con 0x7f041009ef50 2024-09-11T05:26:40.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f0414e84640 1 -- 172.21.15.138:0/3890910647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f041009ef50 msgr2=0x7f041013e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:40.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f0414e84640 1 --2- 172.21.15.138:0/3890910647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f041009ef50 0x7f041013e060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0408009d90 tx=0x7f040801c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e80d6757efc5a79a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 1 -- 172.21.15.138:0/3890910647 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04080ad070 con 0x7f041009ef50 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: dump: 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.996 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: _finish_auth 0 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: _check_auth_tickets 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 1 -- 172.21.15.138:0/3890910647 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04080a9760 con 0x7f041009ef50 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 1 -- 172.21.15.138:0/3890910647 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04080a9b30 con 0x7f041009ef50 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f040effd640 10 monclient: dump: 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:40.997 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 5 monclient: authenticate success, global_id 134155 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:40.998 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 4 set_mon_vals no callback set 2024-09-11T05:26:41.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 10 monclient: shutdown 2024-09-11T05:26:41.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:41.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 -- 172.21.15.138:0/3890910647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f041009ef50 msgr2=0x7f041013e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 --2- 172.21.15.138:0/3890910647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f041009ef50 0x7f041013e060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0408009d90 tx=0x7f040801c830 comp rx=0 tx=0).stop 2024-09-11T05:26:41.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 -- 172.21.15.138:0/3890910647 shutdown_connections 2024-09-11T05:26:41.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 --2- 172.21.15.138:0/3890910647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f041009ef50 0x7f041013e060 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.003 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 -- 172.21.15.138:0/3890910647 >> 172.21.15.138:0/3890910647 conn(0x7f0410063640 msgr2=0x7f0410063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 -- 172.21.15.138:0/3890910647 shutdown_connections 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 -- 172.21.15.138:0/3890910647 wait complete. 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 10 monclient: build_initial_monmap 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 10 monclient: monmap: 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:41.004 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 librados: starting msgr at 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 librados: starting objecter 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 Processor -- start 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 -- start start 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 librados: setting wanted keys 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 1 librados: calling monclient init 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:40.999+0000 7f04168e7640 10 monclient: init 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f041009eb30 0x7f0410134fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient(hunting): picked mon.noname-c con 0x7f041009eb30 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f041009ef50 0x7f04101354e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient(hunting): picked mon.noname-a con 0x7f041009ef50 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:41.005 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04100a32c0 0x7f041013b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient(hunting): picked mon.noname-b con 0x7f04100a32c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f04100a2b40 con 0x7f041009ef50 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f04100a29c0 con 0x7f04100a32c0 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f04100a2cc0 con 0x7f041009eb30 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient(hunting): authenticate will time out at 2084.440183s 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f041009eb30 0x7f0410134fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f041009ef50 0x7f04101354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f041009ef50 0x7f04101354e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59762/0 (socket says 172.21.15.138:59762) 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f041009eb30 0x7f0410134fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55144/0 (socket says 172.21.15.138:55144) 2024-09-11T05:26:41.006 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 -- 172.21.15.138:0/919718686 learned_addr learned my addr 172.21.15.138:0/919718686 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient(hunting): get_auth_request con 0x7f041009ef50 auth_method 0 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 10 monclient(hunting): get_auth_request con 0x7f041009eb30 auth_method 0 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.007 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient(hunting): handle_auth_done global_id 114390 payload 824 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 -- 172.21.15.138:0/919718686 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f041009eb30 msgr2=0x7f0410134fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 --2- 172.21.15.138:0/919718686 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f041009eb30 0x7f0410134fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 -- 172.21.15.138:0/919718686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04100a32c0 msgr2=0x7f041013b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 --2- 172.21.15.138:0/919718686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04100a32c0 0x7f041013b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient: _finish_hunting 0 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 monclient: found mon.noname-a 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 -- 172.21.15.138:0/919718686 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f041006f470 con 0x7f041009ef50 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 -- 172.21.15.138:0/919718686 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f041009ef50 msgr2=0x7f04101354e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040f7fe640 1 --2- 172.21.15.138:0/919718686 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f041009ef50 0x7f04101354e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f040000b410 tx=0x7f040000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 client.?.objecter ms_handle_connect 0x7f041009ef50 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 1 -- 172.21.15.138:0/919718686 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0400015020 con 0x7f041009ef50 2024-09-11T05:26:41.008 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: dump: 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.009 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: _finish_auth 0 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: _check_auth_tickets 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 5 monclient: authenticate success, global_id 114390 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 1 -- 172.21.15.138:0/919718686 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0400011a40 con 0x7f041009ef50 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient: _renew_subs 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f041011ee50 con 0x7f041009ef50 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 1 -- 172.21.15.138:0/919718686 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0400011e30 con 0x7f041009ef50 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040e7fc640 4 set_mon_vals no callback set 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 client.114390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f040cff9640 10 monclient: dump: 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.010 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient: _renew_subs 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f041011dfe0 con 0x7f041009ef50 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 1 librados: init done 2024-09-11T05:26:41.011 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.003+0000 7f04168e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f04168e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f04168e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f041009e420 con 0x7f041009ef50 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040cff9640 1 -- 172.21.15.138:0/919718686 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f040001a400 con 0x7f041009ef50 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040cff9640 1 --2- 172.21.15.138:0/919718686 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03fc03cea0 0x7f03fc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040cff9640 1 -- 172.21.15.138:0/919718686 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f040004f580 con 0x7f041009ef50 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040cff9640 10 client.114390.objecter ms_dispatch 0x7f0410000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040cff9640 3 client.114390.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040cff9640 3 client.114390.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040cff9640 20 client.114390.objecter dump_active .. 0 homeless 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040ffff640 1 --2- 172.21.15.138:0/919718686 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03fc03cea0 0x7f03fc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.012 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040ffff640 10 monclient: get_auth_request con 0x7f03fc03cea0 auth_method 0 2024-09-11T05:26:41.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040cff9640 10 client.114390.objecter ms_handle_connect 0x7f03fc03cea0 2024-09-11T05:26:41.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.007+0000 7f040ffff640 1 --2- 172.21.15.138:0/919718686 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03fc03cea0 0x7f03fc03f360 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f0404007ca0 tx=0x7f0404007bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.011+0000 7f040cff9640 1 -- 172.21.15.138:0/919718686 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f040001f030 con 0x7f041009ef50 2024-09-11T05:26:41.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.011+0000 7f040cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.013 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.011+0000 7f040cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:41.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.171+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f041011d570 con 0x7f03fc03cea0 2024-09-11T05:26:41.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.171+0000 7f040cff9640 1 -- 172.21.15.138:0/919718686 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f041011d570 con 0x7f03fc03cea0 2024-09-11T05:26:41.169 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 4 to deep-scrub 2024-09-11T05:26:41.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 10 librados: watch_flush enter 2024-09-11T05:26:41.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 10 librados: watch_flush exit 2024-09-11T05:26:41.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 20 client.114390.objecter shutdown clearing up homeless session... 2024-09-11T05:26:41.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 10 client.114390.objecter successfully canceled tick 2024-09-11T05:26:41.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03fc03cea0 msgr2=0x7f03fc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 --2- 172.21.15.138:0/919718686 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03fc03cea0 0x7f03fc03f360 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f0404007ca0 tx=0x7f0404007bf0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 10 monclient: shutdown 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f041009ef50 msgr2=0x7f04101354e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 --2- 172.21.15.138:0/919718686 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f041009ef50 0x7f04101354e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f040000b410 tx=0x7f040000b8d0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 shutdown_connections 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 --2- 172.21.15.138:0/919718686 >> v2:172.21.15.190:6800/1562782189 conn(0x7f03fc03cea0 0x7f03fc03f360 unknown :-1 s=CLOSED pgs=3532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 --2- 172.21.15.138:0/919718686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f04100a32c0 0x7f041013b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 --2- 172.21.15.138:0/919718686 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f041009ef50 0x7f04101354e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 --2- 172.21.15.138:0/919718686 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f041009eb30 0x7f0410134fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 >> 172.21.15.138:0/919718686 conn(0x7f0410063640 msgr2=0x7f04100a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 shutdown_connections 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 -- 172.21.15.138:0/919718686 wait complete. 2024-09-11T05:26:41.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.175+0000 7f04168e7640 1 librados: shutdown 2024-09-11T05:26:41.186 INFO:tasks.ceph:Scrubbing osd.5 2024-09-11T05:26:41.187 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-11T05:26:41.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient: get_monmap_and_config 2024-09-11T05:26:41.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient: build_initial_monmap 2024-09-11T05:26:41.308 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient: monmap: 2024-09-11T05:26:41.308 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:41.308 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.308 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:41.314834+0000 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:41.314834+0000 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:41.309 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 1 Processor -- start 2024-09-11T05:26:41.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 1 -- start start 2024-09-11T05:26:41.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient: init 2024-09-11T05:26:41.311 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc12eb40 0x7f97fc12ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient(hunting): picked mon.noname-b con 0x7f97fc12eb40 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f97fc0a30f0 0x7f97fc0a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient(hunting): picked mon.noname-a con 0x7f97fc0a30f0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97fc134e00 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97fc0a29f0 con 0x7f97fc134e00 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f97fc0a2710 con 0x7f97fc0a30f0 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f97fc0a2cd0 con 0x7f97fc12eb40 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f980128c640 10 monclient(hunting): authenticate will time out at 2084.750952s 2024-09-11T05:26:41.312 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97faffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc12eb40 0x7f97fc12ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97faffd640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc12eb40 0x7f97fc12ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55154/0 (socket says 172.21.15.138:55154) 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f97fc0a30f0 0x7f97fc0a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97faffd640 1 -- 172.21.15.138:0/570107084 learned_addr learned my addr 172.21.15.138:0/570107084 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97faffd640 10 monclient(hunting): get_auth_request con 0x7f97fc12eb40 auth_method 0 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97faffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient(hunting): get_auth_request con 0x7f97fc0a30f0 auth_method 0 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.313 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient(hunting): handle_auth_done global_id 114393 payload 274 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 1 -- 172.21.15.138:0/570107084 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc12eb40 msgr2=0x7f97fc12ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 1 --2- 172.21.15.138:0/570107084 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc12eb40 0x7f97fc12ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 1 -- 172.21.15.138:0/570107084 >> v1:172.21.15.190:6789/0 conn(0x7f97fc134e00 legacy=0x7f97fc1352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient: _finish_hunting 0 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 1 monclient: found mon.noname-a 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.314 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.311+0000 7f97fa7fc640 1 -- 172.21.15.138:0/570107084 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc12f460 con 0x7f97fc0a30f0 2024-09-11T05:26:41.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fa7fc640 1 -- 172.21.15.138:0/570107084 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 msgr2=0x7f97fc0a3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:41.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fa7fc640 1 --2- 172.21.15.138:0/570107084 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 0x7f97fc0a3510 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f97f00097c0 tx=0x7f97f001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f7a59289529aaa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 1 -- 172.21.15.138:0/570107084 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00b2070 con 0x7f97fc0a30f0 2024-09-11T05:26:41.316 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: dump: 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: _finish_auth 0 2024-09-11T05:26:41.317 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: _check_auth_tickets 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 1 -- 172.21.15.138:0/570107084 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f00ad640 con 0x7f97fc0a30f0 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 1 -- 172.21.15.138:0/570107084 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00ada10 con 0x7f97fc0a30f0 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 5 monclient: authenticate success, global_id 114393 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 4 set_mon_vals no callback set 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97f9ffb640 10 monclient: dump: 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.318 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient: shutdown 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- 172.21.15.138:0/570107084 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 msgr2=0x7f97fc0a3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 --2- 172.21.15.138:0/570107084 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 0x7f97fc0a3510 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f97f00097c0 tx=0x7f97f001c3f0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- 172.21.15.138:0/570107084 shutdown_connections 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 --2- 172.21.15.138:0/570107084 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 0x7f97fc0a3510 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 --2- 172.21.15.138:0/570107084 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc12eb40 0x7f97fc12ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.319 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- 172.21.15.138:0/570107084 >> 172.21.15.138:0/570107084 conn(0x7f97fc0a7870 msgr2=0x7f97fc0a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- 172.21.15.138:0/570107084 shutdown_connections 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- 172.21.15.138:0/570107084 wait complete. 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient: build_initial_monmap 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient: monmap: 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.320 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 librados: starting msgr at 2024-09-11T05:26:41.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 librados: starting objecter 2024-09-11T05:26:41.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 Processor -- start 2024-09-11T05:26:41.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- start start 2024-09-11T05:26:41.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 librados: setting wanted keys 2024-09-11T05:26:41.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 librados: calling monclient init 2024-09-11T05:26:41.321 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient: init 2024-09-11T05:26:41.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:41.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-11T05:26:41.325 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 0x7f97fc1d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient(hunting): picked mon.noname-a con 0x7f97fc0a30f0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc12eb40 0x7f97fc13f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient(hunting): picked mon.noname-b con 0x7f97fc12eb40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc134e00 0x7f97fc13f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient(hunting): picked mon.noname-c con 0x7f97fc134e00 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc1432b0 con 0x7f97fc0a30f0 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc143130 con 0x7f97fc12eb40 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f97fc143430 con 0x7f97fc134e00 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f980128c640 10 monclient(hunting): authenticate will time out at 2084.758530s 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 0x7f97fc1d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.326 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 0x7f97fc1d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59782/0 (socket says 172.21.15.138:59782) 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fb7fe640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc134e00 0x7f97fc13f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 -- 172.21.15.138:0/3488962795 learned_addr learned my addr 172.21.15.138:0/3488962795 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient(hunting): get_auth_request con 0x7f97fc0a30f0 auth_method 0 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fb7fe640 10 monclient(hunting): get_auth_request con 0x7f97fc134e00 auth_method 0 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.327 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient(hunting): handle_auth_done global_id 114396 payload 824 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 -- 172.21.15.138:0/3488962795 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc134e00 msgr2=0x7f97fc13f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 --2- 172.21.15.138:0/3488962795 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f97fc134e00 0x7f97fc13f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 -- 172.21.15.138:0/3488962795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc12eb40 msgr2=0x7f97fc13f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97fa7fc640 1 --2- 172.21.15.138:0/3488962795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc12eb40 0x7f97fc13f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 --2- 172.21.15.138:0/3488962795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97fc12eb40 0x7f97fc13f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient: _finish_hunting 0 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 monclient: found mon.noname-a 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.319+0000 7f97faffd640 1 -- 172.21.15.138:0/3488962795 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc13ab70 con 0x7f97fc0a30f0 2024-09-11T05:26:41.328 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97faffd640 1 -- 172.21.15.138:0/3488962795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 msgr2=0x7f97fc1d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97faffd640 1 --2- 172.21.15.138:0/3488962795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 0x7f97fc1d0690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f97f400e7e0 tx=0x7f97f400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 client.?.objecter ms_handle_connect 0x7f97fc0a30f0 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4017070 con 0x7f97fc0a30f0 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: dump: 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.329 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: _finish_auth 0 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: _check_auth_tickets 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f400cae0 con 0x7f97fc0a30f0 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 5 monclient: authenticate success, global_id 114396 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 10 monclient: _renew_subs 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97fc09e1f0 con 0x7f97fc0a30f0 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f400f040 con 0x7f97fc0a30f0 2024-09-11T05:26:41.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: dump: 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.331 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97f97fa640 4 set_mon_vals no callback set 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 10 client.114396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 10 monclient: _renew_subs 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97fc13a2d0 con 0x7f97fc0a30f0 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 1 librados: init done 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 10 client.114396.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 15 client.114396.objecter _session_command_op_assign -1 1 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 20 client.114396.objecter _get_session osd=-1 returning homeless 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 10 client.114396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:41.332 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f97fc0a3510 con 0x7f97fc0a30f0 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f97f4012050 con 0x7f97fc0a30f0 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 --2- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97e803cef0 0x7f97e803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f97f404f4d0 con 0x7f97fc0a30f0 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 client.114396.objecter ms_dispatch 0x7f97fc000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 3 client.114396.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 3 client.114396.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 client.114396.objecter checking command 1 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 --2- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6805/189369151 conn(0x7f97e8044450 0x7f97e80468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 20 client.114396.objecter _get_session s=0x7f97e8043da0 osd=5 3 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 20 client.114396.objecter put_session s=0x7f97e8043da0 osd=5 3 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 15 client.114396.objecter _session_command_op_remove -1 1 2024-09-11T05:26:41.333 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 20 client.114396.objecter _get_session s=0x7f97e8043da0 osd=5 3 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 20 client.114396.objecter get_session s=0x7f97e8043da0 osd=5 3 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 15 client.114396.objecter _session_command_op_assign 5 1 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 20 client.114396.objecter put_session s=0x7f97e8043da0 osd=5 4 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 client.114396.objecter _send_command 1 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 --> v2:172.21.15.190:6805/189369151 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97e8046f80 con 0x7f97e8044450 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 20 client.114396.objecter dump_active .. 0 homeless 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_get_version_reply(handle=1 version=698) ==== 24+0+0 (secure 0 0 0) 0x7f97f4056050 con 0x7f97fc0a30f0 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97e37fe640 10 monclient: handle_get_version_reply finishing 1 version 698 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97fa7fc640 1 --2- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97e803cef0 0x7f97e803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97fb7fe640 1 --2- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6805/189369151 conn(0x7f97e8044450 0x7f97e80468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97fa7fc640 10 monclient: get_auth_request con 0x7f97e803cef0 auth_method 0 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.323+0000 7f97fb7fe640 10 monclient: get_auth_request con 0x7f97e8044450 auth_method 0 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.327+0000 7f97fa7fc640 1 --2- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97e803cef0 0x7f97e803f3b0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0a1670 tx=0x7f97f00a7070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.334 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.327+0000 7f97e37fe640 10 client.114396.objecter ms_handle_connect 0x7f97e803cef0 2024-09-11T05:26:41.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.331+0000 7f97e37fe640 10 client.114396.objecter ms_handle_connect 0x7f97e8044450 2024-09-11T05:26:41.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.331+0000 7f97fb7fe640 1 --2- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6805/189369151 conn(0x7f97e8044450 0x7f97e80468f0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.331+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 <== osd.5 v2:172.21.15.190:6805/189369151 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97e8046f80 con 0x7f97e8044450 2024-09-11T05:26:41.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.331+0000 7f97e37fe640 10 client.114396.objecter ms_dispatch 0x7f97fc000ba0 command_reply(tid 1: 0 ) 2024-09-11T05:26:41.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.331+0000 7f97e37fe640 10 client.114396.objecter _finish_command 1 = system:0 2024-09-11T05:26:41.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.331+0000 7f97e37fe640 20 client.114396.objecter put_session s=0x7f97e8043da0 osd=5 4 2024-09-11T05:26:41.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.331+0000 7f97e37fe640 15 client.114396.objecter _session_command_op_remove 5 1 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 10 client.114396.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 15 client.114396.objecter _session_command_op_assign -1 2 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 20 client.114396.objecter _get_session s=0x7f97e8043da0 osd=5 3 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 20 client.114396.objecter put_session s=0x7f97e8043da0 osd=5 3 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 20 client.114396.objecter _get_session s=0x7f97e8043da0 osd=5 3 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 15 client.114396.objecter _session_command_op_remove -1 2 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 20 client.114396.objecter get_session s=0x7f97e8043da0 osd=5 3 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 15 client.114396.objecter _session_command_op_assign 5 2 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 20 client.114396.objecter put_session s=0x7f97e8043da0 osd=5 4 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 10 client.114396.objecter _send_command 2 2024-09-11T05:26:41.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 --> v2:172.21.15.190:6805/189369151 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f97fc136d50 con 0x7f97e8044450 2024-09-11T05:26:41.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f97e37fe640 1 -- 172.21.15.138:0/3488962795 <== osd.5 v2:172.21.15.190:6805/189369151 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f97fc136d50 con 0x7f97e8044450 2024-09-11T05:26:41.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f97e37fe640 10 client.114396.objecter ms_dispatch 0x7f97fc000ba0 command_reply(tid 2: 0 ) 2024-09-11T05:26:41.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f97e37fe640 10 client.114396.objecter _finish_command 2 = system:0 2024-09-11T05:26:41.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f97e37fe640 20 client.114396.objecter put_session s=0x7f97e8043da0 osd=5 4 2024-09-11T05:26:41.342 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f97e37fe640 15 client.114396.objecter _session_command_op_remove 5 2 2024-09-11T05:26:41.342 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-09-11T05:26:41.342 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T05:26:41.342 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-09-11T05:26:41.343 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 10 librados: watch_flush enter 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 10 librados: watch_flush exit 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 10 client.114396.objecter close_session for osd.5 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6805/189369151 conn(0x7f97e8044450 msgr2=0x7f97e80468f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 --2- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6805/189369151 conn(0x7f97e8044450 0x7f97e80468f0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 20 client.114396.objecter put_session s=0x7f97e8043da0 osd=5 1 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 20 client.114396.objecter shutdown clearing up homeless session... 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 10 client.114396.objecter successfully canceled tick 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97e803cef0 msgr2=0x7f97e803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 --2- 172.21.15.138:0/3488962795 >> v2:172.21.15.190:6800/1562782189 conn(0x7f97e803cef0 0x7f97e803f3b0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0a1670 tx=0x7f97f00a7070 comp rx=0 tx=0).stop 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 10 monclient: shutdown 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 msgr2=0x7f97fc1d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.344 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 --2- 172.21.15.138:0/3488962795 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f97fc0a30f0 0x7f97fc1d0690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f97f400e7e0 tx=0x7f97f400eca0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f97fb7fe640 1 -- 172.21.15.138:0/3488962795 reap_dead start 2024-09-11T05:26:41.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 shutdown_connections 2024-09-11T05:26:41.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 >> 172.21.15.138:0/3488962795 conn(0x7f97fc0a7870 msgr2=0x7f97fc134040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:41.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 shutdown_connections 2024-09-11T05:26:41.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 -- 172.21.15.138:0/3488962795 wait complete. 2024-09-11T05:26:41.345 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.343+0000 7f980128c640 1 librados: shutdown 2024-09-11T05:26:41.357 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient: get_monmap_and_config 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient: build_initial_monmap 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient: monmap: 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:41.477605+0000 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:41.477605+0000 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.471 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:41.472 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:41.472 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:41.472 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:41.472 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:41.472 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 1 Processor -- start 2024-09-11T05:26:41.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 1 -- start start 2024-09-11T05:26:41.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient: init 2024-09-11T05:26:41.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:41.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f1158135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient(hunting): picked mon.noname-b con 0x7f1158070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f115809e6e0 0x7f115809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient(hunting): picked mon.noname-a con 0x7f115809e6e0 addr v2:172.21.15.138:3300/0 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f11581382c0 0x7f1158138770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient(hunting): picked mon.noname-c con 0x7f11581382c0 addr v2:172.21.15.190:3300/0 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f11580703f0 con 0x7f115809e6e0 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f11580a8990 con 0x7f11581382c0 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f115809cbf0 con 0x7f1158070010 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115fcdc640 10 monclient(hunting): authenticate will time out at 2084.913700s 2024-09-11T05:26:41.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f115809e6e0 0x7f115809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f1158135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f115809e6e0 0x7f115809eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59786/0 (socket says 172.21.15.138:59786) 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 -- 172.21.15.138:0/3845240126 learned_addr learned my addr 172.21.15.138:0/3845240126 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient(hunting): get_auth_request con 0x7f115809e6e0 auth_method 0 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 10 monclient(hunting): get_auth_request con 0x7f1158070010 auth_method 0 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115e279640 1 --2- 172.21.15.138:0/3845240126 >> v2:172.21.15.190:3300/0 conn(0x7f11581382c0 0x7f1158138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115e279640 10 monclient(hunting): get_auth_request con 0x7f11581382c0 auth_method 0 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115e279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115e279640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115e279640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient(hunting): handle_auth_done global_id 114399 payload 274 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 -- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 msgr2=0x7f1158135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 --2- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f1158135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 -- 172.21.15.138:0/3845240126 >> v2:172.21.15.190:3300/0 conn(0x7f11581382c0 msgr2=0x7f1158138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:41.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 --2- 172.21.15.138:0/3845240126 >> v2:172.21.15.190:3300/0 conn(0x7f11581382c0 0x7f1158138770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient: _finish_hunting 0 2024-09-11T05:26:41.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 monclient: found mon.noname-a 2024-09-11T05:26:41.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:41.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115d277640 1 -- 172.21.15.138:0/3845240126 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115809ced0 con 0x7f115809e6e0 2024-09-11T05:26:41.478 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.475+0000 7f115da78640 1 --2- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f1158135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115d277640 1 -- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f115809e6e0 msgr2=0x7f115809eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115d277640 1 --2- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f115809e6e0 0x7f115809eae0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11580a7a00 tx=0x7f114c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb0c6b7b6cd6d7f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 1 -- 172.21.15.138:0/3845240126 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c0b2070 con 0x7f115809e6e0 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: dump: 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.480 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: _finish_auth 0 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: _check_auth_tickets 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115fcdc640 5 monclient: authenticate success, global_id 114399 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115fcdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 1 -- 172.21.15.138:0/3845240126 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114c0ad640 con 0x7f115809e6e0 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115fcdc640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 1 -- 172.21.15.138:0/3845240126 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c0ada10 con 0x7f115809e6e0 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115fcdc640 4 set_mon_vals no callback set 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.479+0000 7f115ca76640 10 monclient: dump: 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient: shutdown 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f115809e6e0 msgr2=0x7f115809eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 --2- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f115809e6e0 0x7f115809eae0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11580a7a00 tx=0x7f114c01c3b0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- 172.21.15.138:0/3845240126 shutdown_connections 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 --2- 172.21.15.138:0/3845240126 >> v2:172.21.15.190:3300/0 conn(0x7f11581382c0 0x7f1158138770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 --2- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f115809e6e0 0x7f115809eae0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 --2- 172.21.15.138:0/3845240126 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f1158135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- 172.21.15.138:0/3845240126 >> 172.21.15.138:0/3845240126 conn(0x7f11580ab830 msgr2=0x7f11580a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- 172.21.15.138:0/3845240126 shutdown_connections 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- 172.21.15.138:0/3845240126 wait complete. 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient: build_initial_monmap 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient: monmap: 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:41.483 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 librados: starting msgr at 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 librados: starting objecter 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 Processor -- start 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- start start 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 librados: setting wanted keys 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 librados: calling monclient init 2024-09-11T05:26:41.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient: init 2024-09-11T05:26:41.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:41.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:26:41.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f11581d0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient(hunting): picked mon.noname-c con 0x7f1158070010 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:41.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f115809e6e0 0x7f115813e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient(hunting): picked mon.noname-b con 0x7f115809e6e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11581382c0 0x7f115813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient(hunting): picked mon.noname-a con 0x7f11581382c0 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f1158141e70 con 0x7f11581382c0 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1158141ff0 con 0x7f115809e6e0 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f1158142170 con 0x7f1158070010 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115fcdc640 10 monclient(hunting): authenticate will time out at 2084.922019s 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11581382c0 0x7f115813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11581382c0 0x7f115813ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:59788/0 (socket says 172.21.15.138:59788) 2024-09-11T05:26:41.489 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 -- 172.21.15.138:0/25429685 learned_addr learned my addr 172.21.15.138:0/25429685 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient(hunting): get_auth_request con 0x7f11581382c0 auth_method 0 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115da78640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f11581d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115da78640 10 monclient(hunting): get_auth_request con 0x7f1158070010 auth_method 0 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115da78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115da78640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115da78640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115da78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115da78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115da78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115d277640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f115809e6e0 0x7f115813e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient(hunting): handle_auth_done global_id 114402 payload 824 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 -- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 msgr2=0x7f11581d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f11581d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 -- 172.21.15.138:0/25429685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f115809e6e0 msgr2=0x7f115813e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f115809e6e0 0x7f115813e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient: _finish_hunting 0 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 monclient: found mon.noname-a 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115e279640 1 -- 172.21.15.138:0/25429685 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11580a2330 con 0x7f11581382c0 2024-09-11T05:26:41.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115d277640 10 monclient: get_auth_request con 0x7f115809e6e0 auth_method 0 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.483+0000 7f115d277640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f115809e6e0 0x7f115813e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115e279640 1 -- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11581382c0 msgr2=0x7f115813ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115e279640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11581382c0 0x7f115813ecb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f114800d800 tx=0x7f114800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 client.?.objecter ms_handle_connect 0x7f11581382c0 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 1 -- 172.21.15.138:0/25429685 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114800b8d0 con 0x7f11581382c0 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: dump: 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.492 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: _finish_auth 0 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: _check_auth_tickets 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 1 -- 172.21.15.138:0/25429685 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1148019070 con 0x7f11581382c0 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 5 monclient: authenticate success, global_id 114402 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 10 monclient: _renew_subs 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115813aa80 con 0x7f11581382c0 2024-09-11T05:26:41.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 1 -- 172.21.15.138:0/25429685 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114800bcb0 con 0x7f11581382c0 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 monclient: dump: 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.494 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f1157fff640 4 set_mon_vals no callback set 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 10 client.114402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 10 monclient: _renew_subs 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11581d0d60 con 0x7f11581382c0 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 1 librados: init done 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 1 -- 172.21.15.138:0/25429685 <== mon.0 v2:172.21.15.138:3300/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f1148013070 con 0x7f11581382c0 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.138:3300/0 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1158135450 con 0x7f11581382c0 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 1 --2- 172.21.15.138:0/25429685 >> v2:172.21.15.190:6800/1562782189 conn(0x7f114403cf20 0x7f114403f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 1 -- 172.21.15.138:0/25429685 <== mon.0 v2:172.21.15.138:3300/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f114804f7f0 con 0x7f11581382c0 2024-09-11T05:26:41.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 10 client.114402.objecter ms_dispatch 0x7f1158000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 3 client.114402.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 3 client.114402.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f11567fc640 20 client.114402.objecter dump_active .. 0 homeless 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115da78640 1 --2- 172.21.15.138:0/25429685 >> v2:172.21.15.190:6800/1562782189 conn(0x7f114403cf20 0x7f114403f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.487+0000 7f115da78640 10 monclient: get_auth_request con 0x7f114403cf20 auth_method 0 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.491+0000 7f115da78640 1 --2- 172.21.15.138:0/25429685 >> v2:172.21.15.190:6800/1562782189 conn(0x7f114403cf20 0x7f114403f3e0 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7f1150009bd0 tx=0x7f115000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.491+0000 7f11567fc640 10 client.114402.objecter ms_handle_connect 0x7f114403cf20 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.491+0000 7f11567fc640 1 -- 172.21.15.138:0/25429685 <== mon.0 v2:172.21.15.138:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f114804e230 con 0x7f11581382c0 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.491+0000 7f11567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.496 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.491+0000 7f11567fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:41.660 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.663+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f1158139b20 con 0x7f114403cf20 2024-09-11T05:26:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.663+0000 7f11567fc640 1 -- 172.21.15.138:0/25429685 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1158139b20 con 0x7f114403cf20 2024-09-11T05:26:41.661 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 5 to deep-scrub 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 10 librados: watch_flush enter 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 10 librados: watch_flush exit 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 20 client.114402.objecter shutdown clearing up homeless session... 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 10 client.114402.objecter successfully canceled tick 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 >> v2:172.21.15.190:6800/1562782189 conn(0x7f114403cf20 msgr2=0x7f114403f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 --2- 172.21.15.138:0/25429685 >> v2:172.21.15.190:6800/1562782189 conn(0x7f114403cf20 0x7f114403f3e0 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7f1150009bd0 tx=0x7f115000b040 comp rx=0 tx=0).stop 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 10 monclient: shutdown 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11581382c0 msgr2=0x7f115813ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11581382c0 0x7f115813ecb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f114800d800 tx=0x7f114800dcc0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 shutdown_connections 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 --2- 172.21.15.138:0/25429685 >> v2:172.21.15.190:6800/1562782189 conn(0x7f114403cf20 0x7f114403f3e0 unknown :-1 s=CLOSED pgs=3534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11581382c0 0x7f115813ecb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f115809e6e0 0x7f115813e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 --2- 172.21.15.138:0/25429685 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f1158070010 0x7f11581d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 >> 172.21.15.138:0/25429685 conn(0x7f11580ab830 msgr2=0x7f11580a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 shutdown_connections 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 -- 172.21.15.138:0/25429685 wait complete. 2024-09-11T05:26:41.665 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.667+0000 7f115fcdc640 1 librados: shutdown 2024-09-11T05:26:41.680 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:26:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient: get_monmap_and_config 2024-09-11T05:26:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient: build_initial_monmap 2024-09-11T05:26:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient: monmap: 2024-09-11T05:26:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:26:41.807964+0000 2024-09-11T05:26:41.801 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:26:41.807964+0000 2024-09-11T05:26:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:26:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:26:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:26:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:26:41.802 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:26:41.802 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 1 Processor -- start 2024-09-11T05:26:41.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 1 -- start start 2024-09-11T05:26:41.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient: init 2024-09-11T05:26:41.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:41.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-11T05:26:41.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ca00a4030 addr v1:172.21.15.138:6789/0 2024-09-11T05:26:41.804 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4b20 0x7f6ca00a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient(hunting): picked mon.noname-b con 0x7f6ca00a4b20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ca013da40 addr v1:172.21.15.190:6789/0 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.803+0000 7f6ca6cc3640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca00a2850 con 0x7f6ca00a4030 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6ca6cc3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca00a2b30 con 0x7f6ca013da40 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6ca6cc3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6ca6cc3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca00a2e10 con 0x7f6ca00a4b20 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6ca6cc3640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6ca6cc3640 10 monclient(hunting): authenticate will time out at 2085.244021s 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4b20 0x7f6ca00a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4b20 0x7f6ca00a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55210/0 (socket says 172.21.15.138:55210) 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 1 -- 172.21.15.138:0/4149148812 learned_addr learned my addr 172.21.15.138:0/4149148812 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient(hunting): get_auth_request con 0x7f6ca00a4b20 auth_method 0 2024-09-11T05:26:41.805 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6ca4a5f640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f6ca00a4030 0x7f6ca00a4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:34788/0 (socket says 172.21.15.138:34788) 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient(hunting): handle_auth_done global_id 84701 payload 274 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 1 -- 172.21.15.138:0/4149148812 >> v1:172.21.15.190:6789/0 conn(0x7f6ca013da40 legacy=0x7f6ca013fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 1 -- 172.21.15.138:0/4149148812 >> v1:172.21.15.138:6789/0 conn(0x7f6ca00a4030 legacy=0x7f6ca00a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient: _finish_hunting 0 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 1 monclient: found mon.noname-b 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.138:3301/0 2024-09-11T05:26:41.806 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.807+0000 7f6c9ffff640 1 -- 172.21.15.138:0/4149148812 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca00a6c60 con 0x7f6ca00a4b20 2024-09-11T05:26:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9ffff640 1 -- 172.21.15.138:0/4149148812 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4b20 msgr2=0x7f6ca00a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:41.807 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9ffff640 1 --2- 172.21.15.138:0/4149148812 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4b20 0x7f6ca00a3ab0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6c880098a0 tx=0x7f6c8801c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d559424c58c2170 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 1 -- 172.21.15.138:0/4149148812 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c880b0070 con 0x7f6ca00a4b20 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: dump: 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.808 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: _finish_auth 0 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 1 -- 172.21.15.138:0/4149148812 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c880ab600 con 0x7f6ca00a4b20 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 1 -- 172.21.15.138:0/4149148812 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c880ab9d0 con 0x7f6ca00a4b20 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 5 monclient: authenticate success, global_id 84701 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient: get_monmap_and_config success 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 4 set_mon_vals no callback set 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9f7fe640 10 monclient: dump: 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.809 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.810 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.810 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient: shutdown 2024-09-11T05:26:41.814 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/4149148812 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4b20 msgr2=0x7f6ca00a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 --2- 172.21.15.138:0/4149148812 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4b20 0x7f6ca00a3ab0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6c880098a0 tx=0x7f6c8801c380 comp rx=0 tx=0).stop 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/4149148812 shutdown_connections 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 --2- 172.21.15.138:0/4149148812 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4b20 0x7f6ca00a3ab0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/4149148812 >> 172.21.15.138:0/4149148812 conn(0x7f6ca00ab940 msgr2=0x7f6ca00a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/4149148812 shutdown_connections 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/4149148812 wait complete. 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient: build_initial_monmap 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient: monmap: 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.815 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 librados: starting msgr at 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 librados: starting objecter 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 Processor -- start 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- start start 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 librados: setting wanted keys 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 librados: calling monclient init 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient: init 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient: _reopen_session rank -1 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4030 0x7f6ca0124aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca00a4030 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca00a4b20 0x7f6ca013c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.816 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient(hunting): picked mon.noname-b con 0x7f6ca00a4b20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca013da40 0x7f6ca01224d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca013da40 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca01441e0 con 0x7f6ca013da40 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca0144060 con 0x7f6ca00a4b20 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient(hunting): start opening mon connection 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca0144360 con 0x7f6ca00a4030 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient(hunting): _renew_subs 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca6cc3640 10 monclient(hunting): authenticate will time out at 2085.251209s 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4030 0x7f6ca0124aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4030 0x7f6ca0124aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:55224/0 (socket says 172.21.15.138:55224) 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 -- 172.21.15.138:0/688878963 learned_addr learned my addr 172.21.15.138:0/688878963 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient(hunting): get_auth_request con 0x7f6ca00a4030 auth_method 0 2024-09-11T05:26:41.817 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca5260640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca013da40 0x7f6ca01224d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca5260640 10 monclient(hunting): get_auth_request con 0x7f6ca013da40 auth_method 0 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca5260640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca5260640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca5260640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca5260640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca5260640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca5260640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9ffff640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca00a4b20 0x7f6ca013c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.818 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient(hunting): handle_auth_done global_id 84704 payload 824 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 -- 172.21.15.138:0/688878963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca00a4b20 msgr2=0x7f6ca013c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca00a4b20 0x7f6ca013c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 -- 172.21.15.138:0/688878963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca013da40 msgr2=0x7f6ca01224d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca013da40 0x7f6ca01224d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient: _finish_hunting 0 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 monclient: found mon.noname-c 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6ca4a5f640 1 -- 172.21.15.138:0/688878963 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca0125430 con 0x7f6ca00a4030 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9ffff640 10 monclient: get_auth_request con 0x7f6ca00a4b20 auth_method 0 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.811+0000 7f6c9ffff640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca00a4b20 0x7f6ca013c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca4a5f640 1 -- 172.21.15.138:0/688878963 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4030 msgr2=0x7f6ca0124aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca4a5f640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4030 0x7f6ca0124aa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c7f0 tx=0x7f6c9800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 client.?.objecter ms_handle_connect 0x7f6ca00a4030 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:26:41.819 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 1 -- 172.21.15.138:0/688878963 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98017070 con 0x7f6ca00a4030 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: dump: 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: _finish_auth 0 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:26:41.820 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 5 monclient: authenticate success, global_id 84704 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 1 -- 172.21.15.138:0/688878963 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9800a860 con 0x7f6ca00a4030 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 10 monclient: _renew_subs 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca01262a0 con 0x7f6ca00a4030 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 1 -- 172.21.15.138:0/688878963 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800ac50 con 0x7f6ca00a4030 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 monclient: dump: 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:26:41.821 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9effd640 4 set_mon_vals no callback set 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 10 client.84704.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 10 monclient: _renew_subs 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca0123f40 con 0x7f6ca00a4030 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 1 librados: init done 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca00a3ab0 con 0x7f6ca00a4030 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 1 -- 172.21.15.138:0/688878963 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f6c98007070 con 0x7f6ca00a4030 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 1 --2- 172.21.15.138:0/688878963 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c8003ced0 0x7f6c8003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:26:41.822 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 1 -- 172.21.15.138:0/688878963 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f6c9804f180 con 0x7f6ca00a4030 2024-09-11T05:26:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 10 client.84704.objecter ms_dispatch 0x7f6ca0000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:26:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 3 client.84704.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:26:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 3 client.84704.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:26:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9d7fa640 20 client.84704.objecter dump_active .. 0 homeless 2024-09-11T05:26:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9ffff640 1 --2- 172.21.15.138:0/688878963 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c8003ced0 0x7f6c8003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:26:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.815+0000 7f6c9ffff640 10 monclient: get_auth_request con 0x7f6c8003ced0 auth_method 0 2024-09-11T05:26:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.819+0000 7f6c9d7fa640 10 client.84704.objecter ms_handle_connect 0x7f6c8003ced0 2024-09-11T05:26:41.823 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.819+0000 7f6c9ffff640 1 --2- 172.21.15.138:0/688878963 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c8003ced0 0x7f6c8003f390 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0070850 tx=0x7f6c88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:26:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.819+0000 7f6c9d7fa640 1 -- 172.21.15.138:0/688878963 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f6c9801ac60 con 0x7f6ca00a4030 2024-09-11T05:26:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.819+0000 7f6c9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:26:41.824 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.819+0000 7f6c9d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:26:41.985 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.987+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6ca0125f20 con 0x7f6c8003ced0 2024-09-11T05:26:41.990 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6c9d7fa640 1 -- 172.21.15.138:0/688878963 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114117 (secure 0 0 0) 0x7f6ca0125f20 con 0x7f6c8003ced0 2024-09-11T05:26:41.992 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:26:41.992 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:26:41.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 10 librados: watch_flush enter 2024-09-11T05:26:41.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 10 librados: watch_flush exit 2024-09-11T05:26:41.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 20 client.84704.objecter shutdown clearing up homeless session... 2024-09-11T05:26:41.994 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 10 client.84704.objecter successfully canceled tick 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c8003ced0 msgr2=0x7f6c8003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 --2- 172.21.15.138:0/688878963 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c8003ced0 0x7f6c8003f390 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0070850 tx=0x7f6c88008040 comp rx=0 tx=0).stop 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 10 monclient: shutdown 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4030 msgr2=0x7f6ca0124aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4030 0x7f6ca0124aa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c7f0 tx=0x7f6c9800ccb0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 shutdown_connections 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 --2- 172.21.15.138:0/688878963 >> v2:172.21.15.190:6800/1562782189 conn(0x7f6c8003ced0 0x7f6c8003f390 unknown :-1 s=CLOSED pgs=3535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f6ca013da40 0x7f6ca01224d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca00a4b20 0x7f6ca013c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 --2- 172.21.15.138:0/688878963 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f6ca00a4030 0x7f6ca0124aa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:26:41.995 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 >> 172.21.15.138:0/688878963 conn(0x7f6ca00ab940 msgr2=0x7f6ca006f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:26:41.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 shutdown_connections 2024-09-11T05:26:41.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 -- 172.21.15.138:0/688878963 wait complete. 2024-09-11T05:26:41.996 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:26:41.995+0000 7f6ca6cc3640 1 librados: shutdown 2024-09-11T05:26:42.011 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":1263,"stamp":"2024-09-11T05:26:41.063472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":410980,"kb_used_data":21160,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565820060,"statfs":{"total":579820584960,"available":579399741440,"internally_reserved":0,"allocated":21667840,"data_stored":8941820,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":399039137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003687"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1325,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:40.539122+0000","last_change":"2024-09-11T05:26:40.539122+0000","last_active":"2024-09-11T05:26:40.539122+0000","last_peered":"2024-09-11T05:26:40.539122+0000","last_clean":"2024-09-11T05:26:40.539122+0000","last_became_active":"2024-09-11T05:08:46.990401+0000","last_became_peered":"2024-09-11T05:08:46.990401+0000","last_unstale":"2024-09-11T05:26:40.539122+0000","last_undegraded":"2024-09-11T05:26:40.539122+0000","last_fullsized":"2024-09-11T05:26:40.539122+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:40.538988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:40.538988+0000","last_clean_scrub_stamp":"2024-09-11T05:26:40.538988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:38.493633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321517+0000","last_change":"2024-09-11T05:08:46.945587+0000","last_active":"2024-09-11T05:24:50.321517+0000","last_peered":"2024-09-11T05:24:50.321517+0000","last_clean":"2024-09-11T05:24:50.321517+0000","last_became_active":"2024-09-11T05:08:46.944961+0000","last_became_peered":"2024-09-11T05:08:46.944961+0000","last_unstale":"2024-09-11T05:24:50.321517+0000","last_undegraded":"2024-09-11T05:24:50.321517+0000","last_fullsized":"2024-09-11T05:24:50.321517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:51:02.865309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328736+0000","last_change":"2024-09-11T05:08:46.948482+0000","last_active":"2024-09-11T05:24:51.328736+0000","last_peered":"2024-09-11T05:24:51.328736+0000","last_clean":"2024-09-11T05:24:51.328736+0000","last_became_active":"2024-09-11T05:08:46.946203+0000","last_became_peered":"2024-09-11T05:08:46.946203+0000","last_unstale":"2024-09-11T05:24:51.328736+0000","last_undegraded":"2024-09-11T05:24:51.328736+0000","last_fullsized":"2024-09-11T05:24:51.328736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:50.073601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427369+0000","last_change":"2024-09-11T05:08:46.872668+0000","last_active":"2024-09-11T05:24:52.427369+0000","last_peered":"2024-09-11T05:24:52.427369+0000","last_clean":"2024-09-11T05:24:52.427369+0000","last_became_active":"2024-09-11T05:08:46.870538+0000","last_became_peered":"2024-09-11T05:08:46.870538+0000","last_unstale":"2024-09-11T05:24:52.427369+0000","last_undegraded":"2024-09-11T05:24:52.427369+0000","last_fullsized":"2024-09-11T05:24:52.427369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:19:55.699319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443776+0000","last_change":"2024-09-11T05:08:46.869521+0000","last_active":"2024-09-11T05:24:51.443776+0000","last_peered":"2024-09-11T05:24:51.443776+0000","last_clean":"2024-09-11T05:24:51.443776+0000","last_became_active":"2024-09-11T05:08:46.866976+0000","last_became_peered":"2024-09-11T05:08:46.866976+0000","last_unstale":"2024-09-11T05:24:51.443776+0000","last_undegraded":"2024-09-11T05:24:51.443776+0000","last_fullsized":"2024-09-11T05:24:51.443776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:52.479759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943518+0000","last_change":"2024-09-11T05:08:46.990679+0000","last_active":"2024-09-11T05:24:52.943518+0000","last_peered":"2024-09-11T05:24:52.943518+0000","last_clean":"2024-09-11T05:24:52.943518+0000","last_became_active":"2024-09-11T05:08:46.989288+0000","last_became_peered":"2024-09-11T05:08:46.989288+0000","last_unstale":"2024-09-11T05:24:52.943518+0000","last_undegraded":"2024-09-11T05:24:52.943518+0000","last_fullsized":"2024-09-11T05:24:52.943518+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:42.357108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426794+0000","last_change":"2024-09-11T05:08:46.896344+0000","last_active":"2024-09-11T05:24:52.426794+0000","last_peered":"2024-09-11T05:24:52.426794+0000","last_clean":"2024-09-11T05:24:52.426794+0000","last_became_active":"2024-09-11T05:08:46.894236+0000","last_became_peered":"2024-09-11T05:08:46.894236+0000","last_unstale":"2024-09-11T05:24:52.426794+0000","last_undegraded":"2024-09-11T05:24:52.426794+0000","last_fullsized":"2024-09-11T05:24:52.426794+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:30:58.707091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1326,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:39.490756+0000","last_change":"2024-09-11T05:26:39.490756+0000","last_active":"2024-09-11T05:26:39.490756+0000","last_peered":"2024-09-11T05:26:39.490756+0000","last_clean":"2024-09-11T05:26:39.490756+0000","last_became_active":"2024-09-11T05:08:46.925803+0000","last_became_peered":"2024-09-11T05:08:46.925803+0000","last_unstale":"2024-09-11T05:26:39.490756+0000","last_undegraded":"2024-09-11T05:26:39.490756+0000","last_fullsized":"2024-09-11T05:26:39.490756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:39.490554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:39.490554+0000","last_clean_scrub_stamp":"2024-09-11T05:26:39.490554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:28:51.714307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321563+0000","last_change":"2024-09-11T05:08:46.940919+0000","last_active":"2024-09-11T05:24:50.321563+0000","last_peered":"2024-09-11T05:24:50.321563+0000","last_clean":"2024-09-11T05:24:50.321563+0000","last_became_active":"2024-09-11T05:08:46.940655+0000","last_became_peered":"2024-09-11T05:08:46.940655+0000","last_unstale":"2024-09-11T05:24:50.321563+0000","last_undegraded":"2024-09-11T05:24:50.321563+0000","last_fullsized":"2024-09-11T05:24:50.321563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:01:31.299279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328781+0000","last_change":"2024-09-11T05:08:46.995447+0000","last_active":"2024-09-11T05:24:51.328781+0000","last_peered":"2024-09-11T05:24:51.328781+0000","last_clean":"2024-09-11T05:24:51.328781+0000","last_became_active":"2024-09-11T05:08:46.995196+0000","last_became_peered":"2024-09-11T05:08:46.995196+0000","last_unstale":"2024-09-11T05:24:51.328781+0000","last_undegraded":"2024-09-11T05:24:51.328781+0000","last_fullsized":"2024-09-11T05:24:51.328781+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:36:36.230686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943683+0000","last_change":"2024-09-11T05:08:46.900776+0000","last_active":"2024-09-11T05:24:52.943683+0000","last_peered":"2024-09-11T05:24:52.943683+0000","last_clean":"2024-09-11T05:24:52.943683+0000","last_became_active":"2024-09-11T05:08:46.898918+0000","last_became_peered":"2024-09-11T05:08:46.898918+0000","last_unstale":"2024-09-11T05:24:52.943683+0000","last_undegraded":"2024-09-11T05:24:52.943683+0000","last_fullsized":"2024-09-11T05:24:52.943683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:07.016324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444013+0000","last_change":"2024-09-11T05:08:46.882790+0000","last_active":"2024-09-11T05:24:51.444013+0000","last_peered":"2024-09-11T05:24:51.444013+0000","last_clean":"2024-09-11T05:24:51.444013+0000","last_became_active":"2024-09-11T05:08:46.882509+0000","last_became_peered":"2024-09-11T05:08:46.882509+0000","last_unstale":"2024-09-11T05:24:51.444013+0000","last_undegraded":"2024-09-11T05:24:51.444013+0000","last_fullsized":"2024-09-11T05:24:51.444013+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:00:58.227654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444282+0000","last_change":"2024-09-11T05:08:46.906621+0000","last_active":"2024-09-11T05:24:51.444282+0000","last_peered":"2024-09-11T05:24:51.444282+0000","last_clean":"2024-09-11T05:24:51.444282+0000","last_became_active":"2024-09-11T05:08:46.897464+0000","last_became_peered":"2024-09-11T05:08:46.897464+0000","last_unstale":"2024-09-11T05:24:51.444282+0000","last_undegraded":"2024-09-11T05:24:51.444282+0000","last_fullsized":"2024-09-11T05:24:51.444282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:01:20.019858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1426,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426997+0000","last_change":"2024-09-11T05:07:43.495116+0000","last_active":"2024-09-11T05:24:52.426997+0000","last_peered":"2024-09-11T05:24:52.426997+0000","last_clean":"2024-09-11T05:24:52.426997+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:24:52.426997+0000","last_undegraded":"2024-09-11T05:24:52.426997+0000","last_fullsized":"2024-09-11T05:24:52.426997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:41.703163+0000","last_clean_scrub_stamp":"2024-09-11T05:07:41.703163+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:26:01.660249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1336,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443826+0000","last_change":"2024-09-11T05:07:50.540810+0000","last_active":"2024-09-11T05:24:51.443826+0000","last_peered":"2024-09-11T05:24:51.443826+0000","last_clean":"2024-09-11T05:24:51.443826+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:24:51.443826+0000","last_undegraded":"2024-09-11T05:24:51.443826+0000","last_fullsized":"2024-09-11T05:24:51.443826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:32:39.133232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1323,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944078+0000","last_change":"2024-09-11T05:08:46.872094+0000","last_active":"2024-09-11T05:24:52.944078+0000","last_peered":"2024-09-11T05:24:52.944078+0000","last_clean":"2024-09-11T05:24:52.944078+0000","last_became_active":"2024-09-11T05:08:46.869500+0000","last_became_peered":"2024-09-11T05:08:46.869500+0000","last_unstale":"2024-09-11T05:24:52.944078+0000","last_undegraded":"2024-09-11T05:24:52.944078+0000","last_fullsized":"2024-09-11T05:24:52.944078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:06:37.860847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1335,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426715+0000","last_change":"2024-09-11T05:07:50.535448+0000","last_active":"2024-09-11T05:24:52.426715+0000","last_peered":"2024-09-11T05:24:52.426715+0000","last_clean":"2024-09-11T05:24:52.426715+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:24:52.426715+0000","last_undegraded":"2024-09-11T05:24:52.426715+0000","last_fullsized":"2024-09-11T05:24:52.426715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:57.570691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444362+0000","last_change":"2024-09-11T05:08:46.858452+0000","last_active":"2024-09-11T05:24:51.444362+0000","last_peered":"2024-09-11T05:24:51.444362+0000","last_clean":"2024-09-11T05:24:51.444362+0000","last_became_active":"2024-09-11T05:08:46.858029+0000","last_became_peered":"2024-09-11T05:08:46.858029+0000","last_unstale":"2024-09-11T05:24:51.444362+0000","last_undegraded":"2024-09-11T05:24:51.444362+0000","last_fullsized":"2024-09-11T05:24:51.444362+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:29:56.500535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1325,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328504+0000","last_change":"2024-09-11T05:07:50.533338+0000","last_active":"2024-09-11T05:24:51.328504+0000","last_peered":"2024-09-11T05:24:51.328504+0000","last_clean":"2024-09-11T05:24:51.328504+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:24:51.328504+0000","last_undegraded":"2024-09-11T05:24:51.328504+0000","last_fullsized":"2024-09-11T05:24:51.328504+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:08:04.573351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1323,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944250+0000","last_change":"2024-09-11T05:08:46.892894+0000","last_active":"2024-09-11T05:24:52.944250+0000","last_peered":"2024-09-11T05:24:52.944250+0000","last_clean":"2024-09-11T05:24:52.944250+0000","last_became_active":"2024-09-11T05:08:46.892242+0000","last_became_peered":"2024-09-11T05:08:46.892242+0000","last_unstale":"2024-09-11T05:24:52.944250+0000","last_undegraded":"2024-09-11T05:24:52.944250+0000","last_fullsized":"2024-09-11T05:24:52.944250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:52.672673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1337,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.444305+0000","last_change":"2024-09-11T05:07:50.541650+0000","last_active":"2024-09-11T05:24:51.444305+0000","last_peered":"2024-09-11T05:24:51.444305+0000","last_clean":"2024-09-11T05:24:51.444305+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:24:51.444305+0000","last_undegraded":"2024-09-11T05:24:51.444305+0000","last_fullsized":"2024-09-11T05:24:51.444305+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:47.451082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1326,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:38.536043+0000","last_change":"2024-09-11T05:26:38.536043+0000","last_active":"2024-09-11T05:26:38.536043+0000","last_peered":"2024-09-11T05:26:38.536043+0000","last_clean":"2024-09-11T05:26:38.536043+0000","last_became_active":"2024-09-11T05:08:46.907666+0000","last_became_peered":"2024-09-11T05:08:46.907666+0000","last_unstale":"2024-09-11T05:26:38.536043+0000","last_undegraded":"2024-09-11T05:26:38.536043+0000","last_fullsized":"2024-09-11T05:26:38.536043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:38.535970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:38.535970+0000","last_clean_scrub_stamp":"2024-09-11T05:26:38.535970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:04:50.370269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1347,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:39.655639+0000","last_change":"2024-09-11T05:26:39.655639+0000","last_active":"2024-09-11T05:26:39.655639+0000","last_peered":"2024-09-11T05:26:39.655639+0000","last_clean":"2024-09-11T05:26:39.655639+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:26:39.655639+0000","last_undegraded":"2024-09-11T05:26:39.655639+0000","last_fullsized":"2024-09-11T05:26:39.655639+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:39.655493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:39.655493+0000","last_clean_scrub_stamp":"2024-09-11T05:26:39.655493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:34:14.306728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"698'8","reported_seq":1345,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:40.646228+0000","last_change":"2024-09-11T05:26:40.646228+0000","last_active":"2024-09-11T05:26:40.646228+0000","last_peered":"2024-09-11T05:26:40.646228+0000","last_clean":"2024-09-11T05:26:40.646228+0000","last_became_active":"2024-09-11T05:08:46.989555+0000","last_became_peered":"2024-09-11T05:08:46.989555+0000","last_unstale":"2024-09-11T05:26:40.646228+0000","last_undegraded":"2024-09-11T05:26:40.646228+0000","last_fullsized":"2024-09-11T05:26:40.646228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"698'8","last_scrub_stamp":"2024-09-11T05:26:40.646170+0000","last_deep_scrub":"698'8","last_deep_scrub_stamp":"2024-09-11T05:26:40.646170+0000","last_clean_scrub_stamp":"2024-09-11T05:26:40.646170+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:28:04.188890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1335,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427194+0000","last_change":"2024-09-11T05:07:50.535722+0000","last_active":"2024-09-11T05:24:52.427194+0000","last_peered":"2024-09-11T05:24:52.427194+0000","last_clean":"2024-09-11T05:24:52.427194+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:24:52.427194+0000","last_undegraded":"2024-09-11T05:24:52.427194+0000","last_fullsized":"2024-09-11T05:24:52.427194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:07.310720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.426981+0000","last_change":"2024-09-11T05:08:46.919530+0000","last_active":"2024-09-11T05:24:52.426981+0000","last_peered":"2024-09-11T05:24:52.426981+0000","last_clean":"2024-09-11T05:24:52.426981+0000","last_became_active":"2024-09-11T05:08:46.919277+0000","last_became_peered":"2024-09-11T05:08:46.919277+0000","last_unstale":"2024-09-11T05:24:52.426981+0000","last_undegraded":"2024-09-11T05:24:52.426981+0000","last_fullsized":"2024-09-11T05:24:52.426981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:55:31.708446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1338,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.943400+0000","last_change":"2024-09-11T05:07:50.543840+0000","last_active":"2024-09-11T05:24:52.943400+0000","last_peered":"2024-09-11T05:24:52.943400+0000","last_clean":"2024-09-11T05:24:52.943400+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:24:52.943400+0000","last_undegraded":"2024-09-11T05:24:52.943400+0000","last_fullsized":"2024-09-11T05:24:52.943400+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:22:09.809820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1314,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321491+0000","last_change":"2024-09-11T05:08:46.946319+0000","last_active":"2024-09-11T05:24:50.321491+0000","last_peered":"2024-09-11T05:24:50.321491+0000","last_clean":"2024-09-11T05:24:50.321491+0000","last_became_active":"2024-09-11T05:08:46.946035+0000","last_became_peered":"2024-09-11T05:08:46.946035+0000","last_unstale":"2024-09-11T05:24:50.321491+0000","last_undegraded":"2024-09-11T05:24:50.321491+0000","last_fullsized":"2024-09-11T05:24:50.321491+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:58:53.031759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1334,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427259+0000","last_change":"2024-09-11T05:07:50.536048+0000","last_active":"2024-09-11T05:24:52.427259+0000","last_peered":"2024-09-11T05:24:52.427259+0000","last_clean":"2024-09-11T05:24:52.427259+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:24:52.427259+0000","last_undegraded":"2024-09-11T05:24:52.427259+0000","last_fullsized":"2024-09-11T05:24:52.427259+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:07:47.504946+0000","last_clean_scrub_stamp":"2024-09-11T05:07:47.504946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:13:33.527347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328466+0000","last_change":"2024-09-11T05:08:46.928091+0000","last_active":"2024-09-11T05:24:51.328466+0000","last_peered":"2024-09-11T05:24:51.328466+0000","last_clean":"2024-09-11T05:24:51.328466+0000","last_became_active":"2024-09-11T05:08:46.927819+0000","last_became_peered":"2024-09-11T05:08:46.927819+0000","last_unstale":"2024-09-11T05:24:51.328466+0000","last_undegraded":"2024-09-11T05:24:51.328466+0000","last_fullsized":"2024-09-11T05:24:51.328466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:52:50.836005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042863+0000","last_change":"2024-09-11T05:08:47.007949+0000","last_active":"2024-09-11T05:24:50.042863+0000","last_peered":"2024-09-11T05:24:50.042863+0000","last_clean":"2024-09-11T05:24:50.042863+0000","last_became_active":"2024-09-11T05:08:47.006300+0000","last_became_peered":"2024-09-11T05:08:47.006300+0000","last_unstale":"2024-09-11T05:24:50.042863+0000","last_undegraded":"2024-09-11T05:24:50.042863+0000","last_fullsized":"2024-09-11T05:24:50.042863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:29:25.170714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1308,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328753+0000","last_change":"2024-09-11T05:08:46.998761+0000","last_active":"2024-09-11T05:24:51.328753+0000","last_peered":"2024-09-11T05:24:51.328753+0000","last_clean":"2024-09-11T05:24:51.328753+0000","last_became_active":"2024-09-11T05:08:46.998372+0000","last_became_peered":"2024-09-11T05:08:46.998372+0000","last_unstale":"2024-09-11T05:24:51.328753+0000","last_undegraded":"2024-09-11T05:24:51.328753+0000","last_fullsized":"2024-09-11T05:24:51.328753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:14:12.171086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443982+0000","last_change":"2024-09-11T05:08:46.891774+0000","last_active":"2024-09-11T05:24:51.443982+0000","last_peered":"2024-09-11T05:24:51.443982+0000","last_clean":"2024-09-11T05:24:51.443982+0000","last_became_active":"2024-09-11T05:08:46.890326+0000","last_became_peered":"2024-09-11T05:08:46.890326+0000","last_unstale":"2024-09-11T05:24:51.443982+0000","last_undegraded":"2024-09-11T05:24:51.443982+0000","last_fullsized":"2024-09-11T05:24:51.443982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:19.028053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.944186+0000","last_change":"2024-09-11T05:08:46.872093+0000","last_active":"2024-09-11T05:24:52.944186+0000","last_peered":"2024-09-11T05:24:52.944186+0000","last_clean":"2024-09-11T05:24:52.944186+0000","last_became_active":"2024-09-11T05:08:46.866777+0000","last_became_peered":"2024-09-11T05:08:46.866777+0000","last_unstale":"2024-09-11T05:24:52.944186+0000","last_undegraded":"2024-09-11T05:24:52.944186+0000","last_fullsized":"2024-09-11T05:24:52.944186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:15:57.315991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427125+0000","last_change":"2024-09-11T05:08:46.863954+0000","last_active":"2024-09-11T05:24:52.427125+0000","last_peered":"2024-09-11T05:24:52.427125+0000","last_clean":"2024-09-11T05:24:52.427125+0000","last_became_active":"2024-09-11T05:08:46.858140+0000","last_became_peered":"2024-09-11T05:08:46.858140+0000","last_unstale":"2024-09-11T05:24:52.427125+0000","last_undegraded":"2024-09-11T05:24:52.427125+0000","last_fullsized":"2024-09-11T05:24:52.427125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:30.923117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1309,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.328837+0000","last_change":"2024-09-11T05:08:46.945263+0000","last_active":"2024-09-11T05:24:51.328837+0000","last_peered":"2024-09-11T05:24:51.328837+0000","last_clean":"2024-09-11T05:24:51.328837+0000","last_became_active":"2024-09-11T05:08:46.943185+0000","last_became_peered":"2024-09-11T05:08:46.943185+0000","last_unstale":"2024-09-11T05:24:51.328837+0000","last_undegraded":"2024-09-11T05:24:51.328837+0000","last_fullsized":"2024-09-11T05:24:51.328837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:09:14.250455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1319,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:52.427004+0000","last_change":"2024-09-11T05:08:46.921575+0000","last_active":"2024-09-11T05:24:52.427004+0000","last_peered":"2024-09-11T05:24:52.427004+0000","last_clean":"2024-09-11T05:24:52.427004+0000","last_became_active":"2024-09-11T05:08:46.916824+0000","last_became_peered":"2024-09-11T05:08:46.916824+0000","last_unstale":"2024-09-11T05:24:52.427004+0000","last_undegraded":"2024-09-11T05:24:52.427004+0000","last_fullsized":"2024-09-11T05:24:52.427004+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:57.444456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1318,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:51.443671+0000","last_change":"2024-09-11T05:08:46.843485+0000","last_active":"2024-09-11T05:24:51.443671+0000","last_peered":"2024-09-11T05:24:51.443671+0000","last_clean":"2024-09-11T05:24:51.443671+0000","last_became_active":"2024-09-11T05:08:46.843127+0000","last_became_peered":"2024-09-11T05:08:46.843127+0000","last_unstale":"2024-09-11T05:24:51.443671+0000","last_undegraded":"2024-09-11T05:24:51.443671+0000","last_fullsized":"2024-09-11T05:24:51.443671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:50:14.321453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042475+0000","last_change":"2024-09-11T05:08:46.928312+0000","last_active":"2024-09-11T05:24:50.042475+0000","last_peered":"2024-09-11T05:24:50.042475+0000","last_clean":"2024-09-11T05:24:50.042475+0000","last_became_active":"2024-09-11T05:08:46.925739+0000","last_became_peered":"2024-09-11T05:08:46.925739+0000","last_unstale":"2024-09-11T05:24:50.042475+0000","last_undegraded":"2024-09-11T05:24:50.042475+0000","last_fullsized":"2024-09-11T05:24:50.042475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:12.888188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.042613+0000","last_change":"2024-09-11T05:08:46.928128+0000","last_active":"2024-09-11T05:24:50.042613+0000","last_peered":"2024-09-11T05:24:50.042613+0000","last_clean":"2024-09-11T05:24:50.042613+0000","last_became_active":"2024-09-11T05:08:46.925887+0000","last_became_peered":"2024-09-11T05:08:46.925887+0000","last_unstale":"2024-09-11T05:24:50.042613+0000","last_undegraded":"2024-09-11T05:24:50.042613+0000","last_fullsized":"2024-09-11T05:24:50.042613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:27.751280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1313,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:24:50.321512+0000","last_change":"2024-09-11T05:08:46.891878+0000","last_active":"2024-09-11T05:24:50.321512+0000","last_peered":"2024-09-11T05:24:50.321512+0000","last_clean":"2024-09-11T05:24:50.321512+0000","last_became_active":"2024-09-11T05:08:46.889622+0000","last_became_peered":"2024-09-11T05:08:46.889622+0000","last_unstale":"2024-09-11T05:24:50.321512+0000","last_undegraded":"2024-09-11T05:24:50.321512+0000","last_fullsized":"2024-09-11T05:24:50.321512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:08:45.740499+0000","last_clean_scrub_stamp":"2024-09-11T05:08:45.740499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:22.796907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607783,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66696,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305144,"statfs":{"total":96636764160,"available":96568467456,"internally_reserved":0,"allocated":3653632,"data_stored":1842919,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640487,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74952,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296888,"statfs":{"total":96636764160,"available":96560013312,"internally_reserved":0,"allocated":3592192,"data_stored":1314005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640486,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48584,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323256,"statfs":{"total":96636764160,"available":96587014144,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640490,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70792,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301048,"statfs":{"total":96636764160,"available":96564273152,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640487,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66632,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305208,"statfs":{"total":96636764160,"available":96568532992,"internally_reserved":0,"allocated":3596288,"data_stored":1314010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640487,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83324,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94288516,"statfs":{"total":96636764160,"available":96551440384,"internally_reserved":0,"allocated":3649536,"data_stored":1842914,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:26:42.016 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.016 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.017 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.017 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.017 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.017 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.017 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.017 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.017 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.017 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.018 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.018 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T05:07:41.703163+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=41, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.018 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T05:07:47.504946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.018 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.018 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T05:07:47.504946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.018 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.018 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T05:07:47.504946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.018 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T05:07:47.504946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T05:07:47.504946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T05:07:47.504946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T05:07:47.504946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=7, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.019 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-09-11T05:08:45.740499+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=8, tm_sec=45, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=5, tm_min=26, tm_sec=36, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T05:26:42.020 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T05:27:02.022 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T05:27:02.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.150+0000 7f43d6a7a640 10 monclient: get_monmap_and_config 2024-09-11T05:27:02.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.150+0000 7f43d6a7a640 10 monclient: build_initial_monmap 2024-09-11T05:27:02.150 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.150+0000 7f43d6a7a640 10 monclient: monmap: 2024-09-11T05:27:02.150 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:27:02.150 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.150 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:27:02.156670+0000 2024-09-11T05:27:02.150 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:27:02.156670+0000 2024-09-11T05:27:02.150 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:27:02.151 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.151 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:27:02.151 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:27:02.151 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:27:02.151 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:27:02.151 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:27:02.151 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 1 Processor -- start 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 1 -- start start 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient: init 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient: _reopen_session rank -1 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f43d0135490 0x7f43d0073cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient(hunting): picked mon.noname-a con 0x7f43d0135490 addr v2:172.21.15.138:3300/0 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43d0135870 addr v1:172.21.15.138:6789/0 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f43d01367d0 0x7f43d013e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.153 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient(hunting): picked mon.noname-c con 0x7f43d01367d0 addr v2:172.21.15.190:3300/0 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d00a1eb0 con 0x7f43d0135870 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f43d00a7480 con 0x7f43d0135490 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f43d00a6dc0 con 0x7f43d01367d0 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient(hunting): _renew_subs 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d6a7a640 10 monclient(hunting): authenticate will time out at 2105.592816s 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f43d0135490 0x7f43d0073cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43cffff640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f43d0135870 0x7f43d01360c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:56130/0 (socket says 172.21.15.138:56130) 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43cffff640 1 -- 172.21.15.138:0/2027188080 learned_addr learned my addr 172.21.15.138:0/2027188080 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient(hunting): get_auth_request con 0x7f43d0135490 auth_method 0 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:27:02.154 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d5017640 1 --2- 172.21.15.138:0/2027188080 >> v2:172.21.15.190:3300/0 conn(0x7f43d01367d0 0x7f43d013e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient(hunting): handle_auth_done global_id 114408 payload 274 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 1 -- 172.21.15.138:0/2027188080 >> v2:172.21.15.190:3300/0 conn(0x7f43d01367d0 msgr2=0x7f43d013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 1 --2- 172.21.15.138:0/2027188080 >> v2:172.21.15.190:3300/0 conn(0x7f43d01367d0 0x7f43d013e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 1 -- 172.21.15.138:0/2027188080 >> v1:172.21.15.138:6789/0 conn(0x7f43d0135870 legacy=0x7f43d01360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient: _finish_hunting 0 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 1 monclient: found mon.noname-a 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:27:02.155 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.154+0000 7f43d4816640 1 -- 172.21.15.138:0/2027188080 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d0073030 con 0x7f43d0135490 2024-09-11T05:27:02.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d4816640 1 -- 172.21.15.138:0/2027188080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135490 msgr2=0x7f43d0073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:27:02.157 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d4816640 1 --2- 172.21.15.138:0/2027188080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135490 0x7f43d0073cf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f43d0074660 tx=0x7f43c401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d41217f691e517 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 1 -- 172.21.15.138:0/2027188080 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40b0070 con 0x7f43d0135490 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: dump: 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.158 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: _finish_auth 0 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: _check_auth_tickets 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 1 -- 172.21.15.138:0/2027188080 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c40ab600 con 0x7f43d0135490 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 1 -- 172.21.15.138:0/2027188080 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40ab9d0 con 0x7f43d0135490 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:27:02.159 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43cf7fe640 10 monclient: dump: 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d6a7a640 5 monclient: authenticate success, global_id 114408 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d6a7a640 10 monclient: get_monmap_and_config success 2024-09-11T05:27:02.160 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d6a7a640 4 set_mon_vals no callback set 2024-09-11T05:27:02.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d6a7a640 10 monclient: shutdown 2024-09-11T05:27:02.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d6a7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:27:02.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d6a7a640 1 -- 172.21.15.138:0/2027188080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135490 msgr2=0x7f43d0073cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.164 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.158+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/2027188080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135490 0x7f43d0073cf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f43d0074660 tx=0x7f43c401c3d0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 -- 172.21.15.138:0/2027188080 shutdown_connections 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/2027188080 >> v2:172.21.15.190:3300/0 conn(0x7f43d01367d0 0x7f43d013e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/2027188080 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135490 0x7f43d0073cf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 -- 172.21.15.138:0/2027188080 >> 172.21.15.138:0/2027188080 conn(0x7f43d006cc20 msgr2=0x7f43d006d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 -- 172.21.15.138:0/2027188080 shutdown_connections 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 -- 172.21.15.138:0/2027188080 wait complete. 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient: build_initial_monmap 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient: monmap: 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:27:02.165 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 librados: starting msgr at 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 librados: starting objecter 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 Processor -- start 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 -- start start 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 librados: setting wanted keys 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 librados: calling monclient init 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient: init 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient: _reopen_session rank -1 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-11T05:27:02.166 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43d0135490 0x7f43d0137320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient(hunting): picked mon.noname-c con 0x7f43d0135490 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135870 0x7f43d0137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient(hunting): picked mon.noname-a con 0x7f43d0135870 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d01367d0 0x7f43d0137da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient(hunting): picked mon.noname-b con 0x7f43d01367d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f43d0141160 con 0x7f43d0135870 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43d0135eb0 con 0x7f43d01367d0 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f43d01412e0 con 0x7f43d0135490 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient(hunting): _renew_subs 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d6a7a640 10 monclient(hunting): authenticate will time out at 2105.601976s 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135870 0x7f43d0137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43d0135490 0x7f43d0137320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.167 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135870 0x7f43d0137860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3300/0 says I am v2:172.21.15.138:50030/0 (socket says 172.21.15.138:50030) 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43d0135490 0x7f43d0137320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58202/0 (socket says 172.21.15.138:58202) 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 1 -- 172.21.15.138:0/1679928306 learned_addr learned my addr 172.21.15.138:0/1679928306 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 10 monclient(hunting): get_auth_request con 0x7f43d0135870 auth_method 0 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient(hunting): get_auth_request con 0x7f43d0135490 auth_method 0 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:27:02.168 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d5017640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d01367d0 0x7f43d0137da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient(hunting): handle_auth_done global_id 84707 payload 824 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 1 -- 172.21.15.138:0/1679928306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d01367d0 msgr2=0x7f43d0137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d01367d0 0x7f43d0137da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 1 -- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135870 msgr2=0x7f43d0137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135870 0x7f43d0137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient: _finish_hunting 0 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 1 monclient: found mon.noname-c 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d4816640 1 -- 172.21.15.138:0/1679928306 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d013ae20 con 0x7f43d0135490 2024-09-11T05:27:02.169 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43cffff640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135870 0x7f43d0137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d5017640 10 monclient: get_auth_request con 0x7f43d01367d0 auth_method 0 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.162+0000 7f43d5017640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d01367d0 0x7f43d0137da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d4816640 1 -- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43d0135490 msgr2=0x7f43d0137320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d4816640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43d0135490 0x7f43d0137320 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f43d00a71d0 tx=0x7f43c401ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 client.?.objecter ms_handle_connect 0x7f43d0135490 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 1 -- 172.21.15.138:0/1679928306 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40b0050 con 0x7f43d0135490 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: dump: 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.170 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: _finish_auth 0 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: _check_auth_tickets 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 1 -- 172.21.15.138:0/1679928306 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c40bc070 con 0x7f43d0135490 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 5 monclient: authenticate success, global_id 84707 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 10 monclient: _renew_subs 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d00a3640 con 0x7f43d0135490 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T05:27:02.171 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 1 -- 172.21.15.138:0/1679928306 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40b5c80 con 0x7f43d0135490 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 monclient: dump: 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43ce7fc640 4 set_mon_vals no callback set 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 10 client.84707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:27:02.172 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 10 monclient: _renew_subs 2024-09-11T05:27:02.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d013ab40 con 0x7f43d0135490 2024-09-11T05:27:02.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 1 librados: init done 2024-09-11T05:27:02.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:27:02.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:27:02.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d0073cf0 con 0x7f43d0135490 2024-09-11T05:27:02.173 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 1 -- 172.21.15.138:0/1679928306 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f43c40c1460 con 0x7f43d0135490 2024-09-11T05:27:02.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 1 --2- 172.21.15.138:0/1679928306 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43bc03cea0 0x7f43bc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 1 -- 172.21.15.138:0/1679928306 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f43c40ea0c0 con 0x7f43d0135490 2024-09-11T05:27:02.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 10 client.84707.objecter ms_dispatch 0x7f43d0000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:27:02.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 3 client.84707.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:27:02.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 3 client.84707.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:27:02.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cd7fa640 20 client.84707.objecter dump_active .. 0 homeless 2024-09-11T05:27:02.174 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cffff640 1 --2- 172.21.15.138:0/1679928306 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43bc03cea0 0x7f43bc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.166+0000 7f43cffff640 10 monclient: get_auth_request con 0x7f43bc03cea0 auth_method 0 2024-09-11T05:27:02.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.170+0000 7f43cffff640 1 --2- 172.21.15.138:0/1679928306 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43bc03cea0 0x7f43bc03f360 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f43c0007d60 tx=0x7f43c0007cb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:27:02.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.170+0000 7f43cd7fa640 10 client.84707.objecter ms_handle_connect 0x7f43bc03cea0 2024-09-11T05:27:02.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.170+0000 7f43cd7fa640 1 -- 172.21.15.138:0/1679928306 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f43c4016050 con 0x7f43d0135490 2024-09-11T05:27:02.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.170+0000 7f43cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:27:02.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.170+0000 7f43cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:27:02.330 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.330+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 --> v2:172.21.15.190:6800/1562782189 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f43d0139000 con 0x7f43bc03cea0 2024-09-11T05:27:02.335 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43cd7fa640 1 -- 172.21.15.138:0/1679928306 <== mgr.4104 v2:172.21.15.190:6800/1562782189 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114134 (secure 0 0 0) 0x7f43d0139000 con 0x7f43bc03cea0 2024-09-11T05:27:02.337 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-11T05:27:02.338 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2024-09-11T05:27:02.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 10 librados: watch_flush enter 2024-09-11T05:27:02.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 10 librados: watch_flush exit 2024-09-11T05:27:02.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 20 client.84707.objecter shutdown clearing up homeless session... 2024-09-11T05:27:02.339 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 10 client.84707.objecter successfully canceled tick 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43bc03cea0 msgr2=0x7f43bc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/1679928306 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43bc03cea0 0x7f43bc03f360 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f43c0007d60 tx=0x7f43c0007cb0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 10 monclient: shutdown 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43d0135490 msgr2=0x7f43d0137320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43d0135490 0x7f43d0137320 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f43d00a71d0 tx=0x7f43c401ac10 comp rx=0 tx=0).stop 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 shutdown_connections 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/1679928306 >> v2:172.21.15.190:6800/1562782189 conn(0x7f43bc03cea0 0x7f43bc03f360 unknown :-1 s=CLOSED pgs=3536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d01367d0 0x7f43d0137da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f43d0135870 0x7f43d0137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 --2- 172.21.15.138:0/1679928306 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f43d0135490 0x7f43d0137320 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 >> 172.21.15.138:0/1679928306 conn(0x7f43d006cc20 msgr2=0x7f43d00a3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:27:02.340 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 shutdown_connections 2024-09-11T05:27:02.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.338+0000 7f43d6a7a640 1 -- 172.21.15.138:0/1679928306 wait complete. 2024-09-11T05:27:02.341 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.342+0000 7f43d6a7a640 1 librados: shutdown 2024-09-11T05:27:02.354 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":1273,"stamp":"2024-09-11T05:27:01.069772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":410952,"kb_used_data":21160,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565820088,"statfs":{"total":579820584960,"available":579399770112,"internally_reserved":0,"allocated":21667840,"data_stored":8941820,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":399039137},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003780"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1325,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:40.539122+0000","last_change":"2024-09-11T05:26:40.539122+0000","last_active":"2024-09-11T05:26:40.539122+0000","last_peered":"2024-09-11T05:26:40.539122+0000","last_clean":"2024-09-11T05:26:40.539122+0000","last_became_active":"2024-09-11T05:08:46.990401+0000","last_became_peered":"2024-09-11T05:08:46.990401+0000","last_unstale":"2024-09-11T05:26:40.539122+0000","last_undegraded":"2024-09-11T05:26:40.539122+0000","last_fullsized":"2024-09-11T05:26:40.539122+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:40.538988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:40.538988+0000","last_clean_scrub_stamp":"2024-09-11T05:26:40.538988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:38.493633+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1321,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:43.977344+0000","last_change":"2024-09-11T05:26:43.977344+0000","last_active":"2024-09-11T05:26:43.977344+0000","last_peered":"2024-09-11T05:26:43.977344+0000","last_clean":"2024-09-11T05:26:43.977344+0000","last_became_active":"2024-09-11T05:08:46.944961+0000","last_became_peered":"2024-09-11T05:08:46.944961+0000","last_unstale":"2024-09-11T05:26:43.977344+0000","last_undegraded":"2024-09-11T05:26:43.977344+0000","last_fullsized":"2024-09-11T05:26:43.977344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:43.977140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:43.977140+0000","last_clean_scrub_stamp":"2024-09-11T05:26:43.977140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:18:33.229149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1316,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:45.984646+0000","last_change":"2024-09-11T05:26:45.984646+0000","last_active":"2024-09-11T05:26:45.984646+0000","last_peered":"2024-09-11T05:26:45.984646+0000","last_clean":"2024-09-11T05:26:45.984646+0000","last_became_active":"2024-09-11T05:08:46.946203+0000","last_became_peered":"2024-09-11T05:08:46.946203+0000","last_unstale":"2024-09-11T05:26:45.984646+0000","last_undegraded":"2024-09-11T05:26:45.984646+0000","last_fullsized":"2024-09-11T05:26:45.984646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:45.984460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:45.984460+0000","last_clean_scrub_stamp":"2024-09-11T05:26:45.984460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:09:07.422505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1326,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:48.053607+0000","last_change":"2024-09-11T05:26:48.053607+0000","last_active":"2024-09-11T05:26:48.053607+0000","last_peered":"2024-09-11T05:26:48.053607+0000","last_clean":"2024-09-11T05:26:48.053607+0000","last_became_active":"2024-09-11T05:08:46.870538+0000","last_became_peered":"2024-09-11T05:08:46.870538+0000","last_unstale":"2024-09-11T05:26:48.053607+0000","last_undegraded":"2024-09-11T05:26:48.053607+0000","last_fullsized":"2024-09-11T05:26:48.053607+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:48.053446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:48.053446+0000","last_clean_scrub_stamp":"2024-09-11T05:26:48.053446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:49.517605+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1327,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:45.187787+0000","last_change":"2024-09-11T05:26:45.187787+0000","last_active":"2024-09-11T05:26:45.187787+0000","last_peered":"2024-09-11T05:26:45.187787+0000","last_clean":"2024-09-11T05:26:45.187787+0000","last_became_active":"2024-09-11T05:08:46.866976+0000","last_became_peered":"2024-09-11T05:08:46.866976+0000","last_unstale":"2024-09-11T05:26:45.187787+0000","last_undegraded":"2024-09-11T05:26:45.187787+0000","last_fullsized":"2024-09-11T05:26:45.187787+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:45.187617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:45.187617+0000","last_clean_scrub_stamp":"2024-09-11T05:26:45.187617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:41.649714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1331,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:45.656143+0000","last_change":"2024-09-11T05:26:45.656008+0000","last_active":"2024-09-11T05:26:45.656143+0000","last_peered":"2024-09-11T05:26:45.656143+0000","last_clean":"2024-09-11T05:26:45.656143+0000","last_became_active":"2024-09-11T05:08:46.989288+0000","last_became_peered":"2024-09-11T05:08:46.989288+0000","last_unstale":"2024-09-11T05:26:45.656143+0000","last_undegraded":"2024-09-11T05:26:45.656143+0000","last_fullsized":"2024-09-11T05:26:45.656143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:45.655849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:45.655849+0000","last_clean_scrub_stamp":"2024-09-11T05:26:45.655849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:06:20.181026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1327,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:46.042408+0000","last_change":"2024-09-11T05:26:46.042408+0000","last_active":"2024-09-11T05:26:46.042408+0000","last_peered":"2024-09-11T05:26:46.042408+0000","last_clean":"2024-09-11T05:26:46.042408+0000","last_became_active":"2024-09-11T05:08:46.894236+0000","last_became_peered":"2024-09-11T05:08:46.894236+0000","last_unstale":"2024-09-11T05:26:46.042408+0000","last_undegraded":"2024-09-11T05:26:46.042408+0000","last_fullsized":"2024-09-11T05:26:46.042408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:46.042252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:46.042252+0000","last_clean_scrub_stamp":"2024-09-11T05:26:46.042252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:37:50.393296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1326,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:39.490756+0000","last_change":"2024-09-11T05:26:39.490756+0000","last_active":"2024-09-11T05:26:39.490756+0000","last_peered":"2024-09-11T05:26:39.490756+0000","last_clean":"2024-09-11T05:26:39.490756+0000","last_became_active":"2024-09-11T05:08:46.925803+0000","last_became_peered":"2024-09-11T05:08:46.925803+0000","last_unstale":"2024-09-11T05:26:39.490756+0000","last_undegraded":"2024-09-11T05:26:39.490756+0000","last_fullsized":"2024-09-11T05:26:39.490756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:39.490554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:39.490554+0000","last_clean_scrub_stamp":"2024-09-11T05:26:39.490554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:28:51.714307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:42.071131+0000","last_change":"2024-09-11T05:26:42.070968+0000","last_active":"2024-09-11T05:26:42.071131+0000","last_peered":"2024-09-11T05:26:42.071131+0000","last_clean":"2024-09-11T05:26:42.071131+0000","last_became_active":"2024-09-11T05:08:46.940655+0000","last_became_peered":"2024-09-11T05:08:46.940655+0000","last_unstale":"2024-09-11T05:26:42.071131+0000","last_undegraded":"2024-09-11T05:26:42.071131+0000","last_fullsized":"2024-09-11T05:26:42.071131+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:42.070806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:42.070806+0000","last_clean_scrub_stamp":"2024-09-11T05:26:42.070806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:21:25.088765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:43.045811+0000","last_change":"2024-09-11T05:26:43.045674+0000","last_active":"2024-09-11T05:26:43.045811+0000","last_peered":"2024-09-11T05:26:43.045811+0000","last_clean":"2024-09-11T05:26:43.045811+0000","last_became_active":"2024-09-11T05:08:46.995196+0000","last_became_peered":"2024-09-11T05:08:46.995196+0000","last_unstale":"2024-09-11T05:26:43.045811+0000","last_undegraded":"2024-09-11T05:26:43.045811+0000","last_fullsized":"2024-09-11T05:26:43.045811+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:43.045508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:43.045508+0000","last_clean_scrub_stamp":"2024-09-11T05:26:43.045508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:54:24.438028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1331,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:44.684224+0000","last_change":"2024-09-11T05:26:44.684003+0000","last_active":"2024-09-11T05:26:44.684224+0000","last_peered":"2024-09-11T05:26:44.684224+0000","last_clean":"2024-09-11T05:26:44.684224+0000","last_became_active":"2024-09-11T05:08:46.898918+0000","last_became_peered":"2024-09-11T05:08:46.898918+0000","last_unstale":"2024-09-11T05:26:44.684224+0000","last_undegraded":"2024-09-11T05:26:44.684224+0000","last_fullsized":"2024-09-11T05:26:44.684224+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:44.683818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:44.683818+0000","last_clean_scrub_stamp":"2024-09-11T05:26:44.683818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:03.974717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1327,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:46.236031+0000","last_change":"2024-09-11T05:26:46.235888+0000","last_active":"2024-09-11T05:26:46.236031+0000","last_peered":"2024-09-11T05:26:46.236031+0000","last_clean":"2024-09-11T05:26:46.236031+0000","last_became_active":"2024-09-11T05:08:46.882509+0000","last_became_peered":"2024-09-11T05:08:46.882509+0000","last_unstale":"2024-09-11T05:26:46.236031+0000","last_undegraded":"2024-09-11T05:26:46.236031+0000","last_fullsized":"2024-09-11T05:26:46.236031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:46.235736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:46.235736+0000","last_clean_scrub_stamp":"2024-09-11T05:26:46.235736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:54:15.814349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1327,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:49.193684+0000","last_change":"2024-09-11T05:26:49.193684+0000","last_active":"2024-09-11T05:26:49.193684+0000","last_peered":"2024-09-11T05:26:49.193684+0000","last_clean":"2024-09-11T05:26:49.193684+0000","last_became_active":"2024-09-11T05:08:46.897464+0000","last_became_peered":"2024-09-11T05:08:46.897464+0000","last_unstale":"2024-09-11T05:26:49.193684+0000","last_undegraded":"2024-09-11T05:26:49.193684+0000","last_fullsized":"2024-09-11T05:26:49.193684+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:49.193516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:49.193516+0000","last_clean_scrub_stamp":"2024-09-11T05:26:49.193516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:12:51.961354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"698'74","reported_seq":1436,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:40.993241+0000","last_change":"2024-09-11T05:26:40.993241+0000","last_active":"2024-09-11T05:26:40.993241+0000","last_peered":"2024-09-11T05:26:40.993241+0000","last_clean":"2024-09-11T05:26:40.993241+0000","last_became_active":"2024-09-11T05:07:43.494234+0000","last_became_peered":"2024-09-11T05:07:43.494234+0000","last_unstale":"2024-09-11T05:26:40.993241+0000","last_undegraded":"2024-09-11T05:26:40.993241+0000","last_fullsized":"2024-09-11T05:26:40.993241+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"698'74","last_scrub_stamp":"2024-09-11T05:26:40.993059+0000","last_deep_scrub":"698'74","last_deep_scrub_stamp":"2024-09-11T05:26:40.993059+0000","last_clean_scrub_stamp":"2024-09-11T05:26:40.993059+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:50:45.953897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"698'2","reported_seq":1345,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:43.250997+0000","last_change":"2024-09-11T05:26:43.250997+0000","last_active":"2024-09-11T05:26:43.250997+0000","last_peered":"2024-09-11T05:26:43.250997+0000","last_clean":"2024-09-11T05:26:43.250997+0000","last_became_active":"2024-09-11T05:07:48.534307+0000","last_became_peered":"2024-09-11T05:07:48.534307+0000","last_unstale":"2024-09-11T05:26:43.250997+0000","last_undegraded":"2024-09-11T05:26:43.250997+0000","last_fullsized":"2024-09-11T05:26:43.250997+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"698'2","last_scrub_stamp":"2024-09-11T05:26:43.250838+0000","last_deep_scrub":"698'2","last_deep_scrub_stamp":"2024-09-11T05:26:43.250838+0000","last_clean_scrub_stamp":"2024-09-11T05:26:43.250838+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:05:41.296356+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00092906,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1331,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:41.646680+0000","last_change":"2024-09-11T05:26:41.646680+0000","last_active":"2024-09-11T05:26:41.646680+0000","last_peered":"2024-09-11T05:26:41.646680+0000","last_clean":"2024-09-11T05:26:41.646680+0000","last_became_active":"2024-09-11T05:08:46.869500+0000","last_became_peered":"2024-09-11T05:08:46.869500+0000","last_unstale":"2024-09-11T05:26:41.646680+0000","last_undegraded":"2024-09-11T05:26:41.646680+0000","last_fullsized":"2024-09-11T05:26:41.646680+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:41.646526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:41.646526+0000","last_clean_scrub_stamp":"2024-09-11T05:26:41.646526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:11:01.003840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1343,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:43.968285+0000","last_change":"2024-09-11T05:26:43.968285+0000","last_active":"2024-09-11T05:26:43.968285+0000","last_peered":"2024-09-11T05:26:43.968285+0000","last_clean":"2024-09-11T05:26:43.968285+0000","last_became_active":"2024-09-11T05:07:48.528831+0000","last_became_peered":"2024-09-11T05:07:48.528831+0000","last_unstale":"2024-09-11T05:26:43.968285+0000","last_undegraded":"2024-09-11T05:26:43.968285+0000","last_fullsized":"2024-09-11T05:26:43.968285+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:43.968125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:43.968125+0000","last_clean_scrub_stamp":"2024-09-11T05:26:43.968125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:57:02.913485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082934999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1326,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:44.222585+0000","last_change":"2024-09-11T05:26:44.222585+0000","last_active":"2024-09-11T05:26:44.222585+0000","last_peered":"2024-09-11T05:26:44.222585+0000","last_clean":"2024-09-11T05:26:44.222585+0000","last_became_active":"2024-09-11T05:08:46.858029+0000","last_became_peered":"2024-09-11T05:08:46.858029+0000","last_unstale":"2024-09-11T05:26:44.222585+0000","last_undegraded":"2024-09-11T05:26:44.222585+0000","last_fullsized":"2024-09-11T05:26:44.222585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:44.222433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:44.222433+0000","last_clean_scrub_stamp":"2024-09-11T05:26:44.222433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:07.835610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1333,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:40.984330+0000","last_change":"2024-09-11T05:26:40.984330+0000","last_active":"2024-09-11T05:26:40.984330+0000","last_peered":"2024-09-11T05:26:40.984330+0000","last_clean":"2024-09-11T05:26:40.984330+0000","last_became_active":"2024-09-11T05:07:48.527601+0000","last_became_peered":"2024-09-11T05:07:48.527601+0000","last_unstale":"2024-09-11T05:26:40.984330+0000","last_undegraded":"2024-09-11T05:26:40.984330+0000","last_fullsized":"2024-09-11T05:26:40.984330+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:40.984268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:40.984268+0000","last_clean_scrub_stamp":"2024-09-11T05:26:40.984268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:36:13.354459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011137250000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1331,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:43.695570+0000","last_change":"2024-09-11T05:26:43.695570+0000","last_active":"2024-09-11T05:26:43.695570+0000","last_peered":"2024-09-11T05:26:43.695570+0000","last_clean":"2024-09-11T05:26:43.695570+0000","last_became_active":"2024-09-11T05:08:46.892242+0000","last_became_peered":"2024-09-11T05:08:46.892242+0000","last_unstale":"2024-09-11T05:26:43.695570+0000","last_undegraded":"2024-09-11T05:26:43.695570+0000","last_fullsized":"2024-09-11T05:26:43.695570+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:43.695411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:43.695411+0000","last_clean_scrub_stamp":"2024-09-11T05:26:43.695411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:12.838987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"698'3","reported_seq":1346,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:42.247844+0000","last_change":"2024-09-11T05:26:42.247844+0000","last_active":"2024-09-11T05:26:42.247844+0000","last_peered":"2024-09-11T05:26:42.247844+0000","last_clean":"2024-09-11T05:26:42.247844+0000","last_became_active":"2024-09-11T05:07:48.532998+0000","last_became_peered":"2024-09-11T05:07:48.532998+0000","last_unstale":"2024-09-11T05:26:42.247844+0000","last_undegraded":"2024-09-11T05:26:42.247844+0000","last_fullsized":"2024-09-11T05:26:42.247844+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"698'3","last_scrub_stamp":"2024-09-11T05:26:42.247555+0000","last_deep_scrub":"698'3","last_deep_scrub_stamp":"2024-09-11T05:26:42.247555+0000","last_clean_scrub_stamp":"2024-09-11T05:26:42.247555+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:22:43.354966+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001908505,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1327,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:38.536091+0000","last_change":"2024-09-11T05:26:38.536043+0000","last_active":"2024-09-11T05:26:38.536091+0000","last_peered":"2024-09-11T05:26:38.536091+0000","last_clean":"2024-09-11T05:26:38.536091+0000","last_became_active":"2024-09-11T05:08:46.907666+0000","last_became_peered":"2024-09-11T05:08:46.907666+0000","last_unstale":"2024-09-11T05:26:38.536091+0000","last_undegraded":"2024-09-11T05:26:38.536091+0000","last_fullsized":"2024-09-11T05:26:38.536091+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:38.535970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:38.535970+0000","last_clean_scrub_stamp":"2024-09-11T05:26:38.535970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:04:50.370269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1347,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:39.655639+0000","last_change":"2024-09-11T05:26:39.655639+0000","last_active":"2024-09-11T05:26:39.655639+0000","last_peered":"2024-09-11T05:26:39.655639+0000","last_clean":"2024-09-11T05:26:39.655639+0000","last_became_active":"2024-09-11T05:07:48.522530+0000","last_became_peered":"2024-09-11T05:07:48.522530+0000","last_unstale":"2024-09-11T05:26:39.655639+0000","last_undegraded":"2024-09-11T05:26:39.655639+0000","last_fullsized":"2024-09-11T05:26:39.655639+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:39.655493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:39.655493+0000","last_clean_scrub_stamp":"2024-09-11T05:26:39.655493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:34:14.306728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012341209999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"698'8","reported_seq":1345,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:40.646228+0000","last_change":"2024-09-11T05:26:40.646228+0000","last_active":"2024-09-11T05:26:40.646228+0000","last_peered":"2024-09-11T05:26:40.646228+0000","last_clean":"2024-09-11T05:26:40.646228+0000","last_became_active":"2024-09-11T05:08:46.989555+0000","last_became_peered":"2024-09-11T05:08:46.989555+0000","last_unstale":"2024-09-11T05:26:40.646228+0000","last_undegraded":"2024-09-11T05:26:40.646228+0000","last_fullsized":"2024-09-11T05:26:40.646228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"698'8","last_scrub_stamp":"2024-09-11T05:26:40.646170+0000","last_deep_scrub":"698'8","last_deep_scrub_stamp":"2024-09-11T05:26:40.646170+0000","last_clean_scrub_stamp":"2024-09-11T05:26:40.646170+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:28:04.188890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1343,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:41.988386+0000","last_change":"2024-09-11T05:26:41.988386+0000","last_active":"2024-09-11T05:26:41.988386+0000","last_peered":"2024-09-11T05:26:41.988386+0000","last_clean":"2024-09-11T05:26:41.988386+0000","last_became_active":"2024-09-11T05:07:48.530746+0000","last_became_peered":"2024-09-11T05:07:48.530746+0000","last_unstale":"2024-09-11T05:26:41.988386+0000","last_undegraded":"2024-09-11T05:26:41.988386+0000","last_fullsized":"2024-09-11T05:26:41.988386+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:41.988225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:41.988225+0000","last_clean_scrub_stamp":"2024-09-11T05:26:41.988225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:02:27.451712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010691349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1327,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:43.009674+0000","last_change":"2024-09-11T05:26:43.009674+0000","last_active":"2024-09-11T05:26:43.009674+0000","last_peered":"2024-09-11T05:26:43.009674+0000","last_clean":"2024-09-11T05:26:43.009674+0000","last_became_active":"2024-09-11T05:08:46.919277+0000","last_became_peered":"2024-09-11T05:08:46.919277+0000","last_unstale":"2024-09-11T05:26:43.009674+0000","last_undegraded":"2024-09-11T05:26:43.009674+0000","last_fullsized":"2024-09-11T05:26:43.009674+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:43.009518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:43.009518+0000","last_clean_scrub_stamp":"2024-09-11T05:26:43.009518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:48:09.658318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1346,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:42.664841+0000","last_change":"2024-09-11T05:26:42.664841+0000","last_active":"2024-09-11T05:26:42.664841+0000","last_peered":"2024-09-11T05:26:42.664841+0000","last_clean":"2024-09-11T05:26:42.664841+0000","last_became_active":"2024-09-11T05:07:48.529371+0000","last_became_peered":"2024-09-11T05:07:48.529371+0000","last_unstale":"2024-09-11T05:26:42.664841+0000","last_undegraded":"2024-09-11T05:26:42.664841+0000","last_fullsized":"2024-09-11T05:26:42.664841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:42.664690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:42.664690+0000","last_clean_scrub_stamp":"2024-09-11T05:26:42.664690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:46:29.759321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141553,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1322,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:43.024761+0000","last_change":"2024-09-11T05:26:43.024761+0000","last_active":"2024-09-11T05:26:43.024761+0000","last_peered":"2024-09-11T05:26:43.024761+0000","last_clean":"2024-09-11T05:26:43.024761+0000","last_became_active":"2024-09-11T05:08:46.946035+0000","last_became_peered":"2024-09-11T05:08:46.946035+0000","last_unstale":"2024-09-11T05:26:43.024761+0000","last_undegraded":"2024-09-11T05:26:43.024761+0000","last_fullsized":"2024-09-11T05:26:43.024761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:43.024603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:43.024603+0000","last_clean_scrub_stamp":"2024-09-11T05:26:43.024603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:35:44.330742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1342,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:45.010050+0000","last_change":"2024-09-11T05:26:45.010050+0000","last_active":"2024-09-11T05:26:45.010050+0000","last_peered":"2024-09-11T05:26:45.010050+0000","last_clean":"2024-09-11T05:26:45.010050+0000","last_became_active":"2024-09-11T05:07:48.578050+0000","last_became_peered":"2024-09-11T05:07:48.578050+0000","last_unstale":"2024-09-11T05:26:45.010050+0000","last_undegraded":"2024-09-11T05:26:45.010050+0000","last_fullsized":"2024-09-11T05:26:45.010050+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:45.009889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:45.009889+0000","last_clean_scrub_stamp":"2024-09-11T05:26:45.009889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:33:14.554536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013491429999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1316,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:44.018783+0000","last_change":"2024-09-11T05:26:44.018783+0000","last_active":"2024-09-11T05:26:44.018783+0000","last_peered":"2024-09-11T05:26:44.018783+0000","last_clean":"2024-09-11T05:26:44.018783+0000","last_became_active":"2024-09-11T05:08:46.927819+0000","last_became_peered":"2024-09-11T05:08:46.927819+0000","last_unstale":"2024-09-11T05:26:44.018783+0000","last_undegraded":"2024-09-11T05:26:44.018783+0000","last_fullsized":"2024-09-11T05:26:44.018783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:44.018617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:44.018617+0000","last_clean_scrub_stamp":"2024-09-11T05:26:44.018617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:57:22.801171+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1325,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:43.615533+0000","last_change":"2024-09-11T05:26:43.615533+0000","last_active":"2024-09-11T05:26:43.615533+0000","last_peered":"2024-09-11T05:26:43.615533+0000","last_clean":"2024-09-11T05:26:43.615533+0000","last_became_active":"2024-09-11T05:08:47.006300+0000","last_became_peered":"2024-09-11T05:08:47.006300+0000","last_unstale":"2024-09-11T05:26:43.615533+0000","last_undegraded":"2024-09-11T05:26:43.615533+0000","last_fullsized":"2024-09-11T05:26:43.615533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:43.615371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:43.615371+0000","last_clean_scrub_stamp":"2024-09-11T05:26:43.615371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:05:28.973432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1316,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:42.005186+0000","last_change":"2024-09-11T05:26:42.005186+0000","last_active":"2024-09-11T05:26:42.005186+0000","last_peered":"2024-09-11T05:26:42.005186+0000","last_clean":"2024-09-11T05:26:42.005186+0000","last_became_active":"2024-09-11T05:08:46.998372+0000","last_became_peered":"2024-09-11T05:08:46.998372+0000","last_unstale":"2024-09-11T05:26:42.005186+0000","last_undegraded":"2024-09-11T05:26:42.005186+0000","last_fullsized":"2024-09-11T05:26:42.005186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:42.005113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:42.005113+0000","last_clean_scrub_stamp":"2024-09-11T05:26:42.005113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:13:14.140869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1326,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:47.231459+0000","last_change":"2024-09-11T05:26:47.231459+0000","last_active":"2024-09-11T05:26:47.231459+0000","last_peered":"2024-09-11T05:26:47.231459+0000","last_clean":"2024-09-11T05:26:47.231459+0000","last_became_active":"2024-09-11T05:08:46.890326+0000","last_became_peered":"2024-09-11T05:08:46.890326+0000","last_unstale":"2024-09-11T05:26:47.231459+0000","last_undegraded":"2024-09-11T05:26:47.231459+0000","last_fullsized":"2024-09-11T05:26:47.231459+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:47.231303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:47.231303+0000","last_clean_scrub_stamp":"2024-09-11T05:26:47.231303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:59:52.195162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1330,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:46.645184+0000","last_change":"2024-09-11T05:26:46.645184+0000","last_active":"2024-09-11T05:26:46.645184+0000","last_peered":"2024-09-11T05:26:46.645184+0000","last_clean":"2024-09-11T05:26:46.645184+0000","last_became_active":"2024-09-11T05:08:46.866777+0000","last_became_peered":"2024-09-11T05:08:46.866777+0000","last_unstale":"2024-09-11T05:26:46.645184+0000","last_undegraded":"2024-09-11T05:26:46.645184+0000","last_fullsized":"2024-09-11T05:26:46.645184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:46.645021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:46.645021+0000","last_clean_scrub_stamp":"2024-09-11T05:26:46.645021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:22.140303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1326,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:47.010554+0000","last_change":"2024-09-11T05:26:47.010554+0000","last_active":"2024-09-11T05:26:47.010554+0000","last_peered":"2024-09-11T05:26:47.010554+0000","last_clean":"2024-09-11T05:26:47.010554+0000","last_became_active":"2024-09-11T05:08:46.858140+0000","last_became_peered":"2024-09-11T05:08:46.858140+0000","last_unstale":"2024-09-11T05:26:47.010554+0000","last_undegraded":"2024-09-11T05:26:47.010554+0000","last_fullsized":"2024-09-11T05:26:47.010554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:47.010395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:47.010395+0000","last_clean_scrub_stamp":"2024-09-11T05:26:47.010395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:02:54.699252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1317,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:44.975287+0000","last_change":"2024-09-11T05:26:44.975287+0000","last_active":"2024-09-11T05:26:44.975287+0000","last_peered":"2024-09-11T05:26:44.975287+0000","last_clean":"2024-09-11T05:26:44.975287+0000","last_became_active":"2024-09-11T05:08:46.943185+0000","last_became_peered":"2024-09-11T05:08:46.943185+0000","last_unstale":"2024-09-11T05:26:44.975287+0000","last_undegraded":"2024-09-11T05:26:44.975287+0000","last_fullsized":"2024-09-11T05:26:44.975287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:44.975125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:44.975125+0000","last_clean_scrub_stamp":"2024-09-11T05:26:44.975125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:07:33.066332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1328,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:49.079571+0000","last_change":"2024-09-11T05:26:49.079571+0000","last_active":"2024-09-11T05:26:49.079571+0000","last_peered":"2024-09-11T05:26:49.079571+0000","last_clean":"2024-09-11T05:26:49.079571+0000","last_became_active":"2024-09-11T05:08:46.916824+0000","last_became_peered":"2024-09-11T05:08:46.916824+0000","last_unstale":"2024-09-11T05:26:49.079571+0000","last_undegraded":"2024-09-11T05:26:49.079571+0000","last_fullsized":"2024-09-11T05:26:49.079571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:49.079412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:49.079412+0000","last_clean_scrub_stamp":"2024-09-11T05:26:49.079412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:14:52.093584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1326,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:48.213808+0000","last_change":"2024-09-11T05:26:48.213808+0000","last_active":"2024-09-11T05:26:48.213808+0000","last_peered":"2024-09-11T05:26:48.213808+0000","last_clean":"2024-09-11T05:26:48.213808+0000","last_became_active":"2024-09-11T05:08:46.843127+0000","last_became_peered":"2024-09-11T05:08:46.843127+0000","last_unstale":"2024-09-11T05:26:48.213808+0000","last_undegraded":"2024-09-11T05:26:48.213808+0000","last_fullsized":"2024-09-11T05:26:48.213808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:48.213627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:48.213627+0000","last_clean_scrub_stamp":"2024-09-11T05:26:48.213627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:45.555792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1325,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:41.542099+0000","last_change":"2024-09-11T05:26:41.542099+0000","last_active":"2024-09-11T05:26:41.542099+0000","last_peered":"2024-09-11T05:26:41.542099+0000","last_clean":"2024-09-11T05:26:41.542099+0000","last_became_active":"2024-09-11T05:08:46.925739+0000","last_became_peered":"2024-09-11T05:08:46.925739+0000","last_unstale":"2024-09-11T05:26:41.542099+0000","last_undegraded":"2024-09-11T05:26:41.542099+0000","last_fullsized":"2024-09-11T05:26:41.542099+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:41.541978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:41.541978+0000","last_clean_scrub_stamp":"2024-09-11T05:26:41.541978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:17:22.748680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1325,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:42.565416+0000","last_change":"2024-09-11T05:26:42.565416+0000","last_active":"2024-09-11T05:26:42.565416+0000","last_peered":"2024-09-11T05:26:42.565416+0000","last_clean":"2024-09-11T05:26:42.565416+0000","last_became_active":"2024-09-11T05:08:46.925887+0000","last_became_peered":"2024-09-11T05:08:46.925887+0000","last_unstale":"2024-09-11T05:26:42.565416+0000","last_undegraded":"2024-09-11T05:26:42.565416+0000","last_fullsized":"2024-09-11T05:26:42.565416+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:42.565216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:42.565216+0000","last_clean_scrub_stamp":"2024-09-11T05:26:42.565216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:09:16.543433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1321,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-11T05:26:44.961447+0000","last_change":"2024-09-11T05:26:44.961447+0000","last_active":"2024-09-11T05:26:44.961447+0000","last_peered":"2024-09-11T05:26:44.961447+0000","last_clean":"2024-09-11T05:26:44.961447+0000","last_became_active":"2024-09-11T05:08:46.889622+0000","last_became_peered":"2024-09-11T05:08:46.889622+0000","last_unstale":"2024-09-11T05:26:44.961447+0000","last_undegraded":"2024-09-11T05:26:44.961447+0000","last_fullsized":"2024-09-11T05:26:44.961447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T05:26:44.961258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T05:26:44.961258+0000","last_clean_scrub_stamp":"2024-09-11T05:26:44.961258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:11:03.322233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60698,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66688,"kb_used_data":3568,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305152,"statfs":{"total":96636764160,"available":96568475648,"internally_reserved":0,"allocated":3653632,"data_stored":1842919,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640491,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74948,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296892,"statfs":{"total":96636764160,"available":96560017408,"internally_reserved":0,"allocated":3592192,"data_stored":1314005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640490,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48576,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323264,"statfs":{"total":96636764160,"available":96587022336,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640494,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70784,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301056,"statfs":{"total":96636764160,"available":96564281344,"internally_reserved":0,"allocated":3588096,"data_stored":1313986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66632,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305208,"statfs":{"total":96636764160,"available":96568532992,"internally_reserved":0,"allocated":3596288,"data_stored":1314010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83324,"kb_used_data":3564,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94288516,"statfs":{"total":96636764160,"available":96551440384,"internally_reserved":0,"allocated":3649536,"data_stored":1842914,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30349,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T05:27:02.357 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T05:27:02.470 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.470+0000 7f11ff5e8640 10 monclient: get_monmap_and_config 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.470+0000 7f11ff5e8640 10 monclient: build_initial_monmap 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.470+0000 7f11ff5e8640 10 monclient: monmap: 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:27:02.477028+0000 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:27:02.477028+0000 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:0: v1:172.21.15.138:6789/0 mon.noname-a-legacy 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c-legacy 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-b 2024-09-11T05:27:02.471 INFO:teuthology.orchestra.run.smithi138.stderr:3: v2:172.21.15.138:3300/0 mon.noname-a 2024-09-11T05:27:02.472 INFO:teuthology.orchestra.run.smithi138.stderr:4: v2:172.21.15.190:3300/0 mon.noname-c 2024-09-11T05:27:02.472 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.473 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 1 Processor -- start 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 1 -- start start 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient: init 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient: _reopen_session rank -1 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f11f809e500 0x7f11f809e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient(hunting): picked mon.noname-a con 0x7f11f809e500 addr v2:172.21.15.138:3300/0 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11f80a1b20 addr v1:172.21.15.138:6789/0 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809ee20 0x7f11f8135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient(hunting): picked mon.noname-b con 0x7f11f809ee20 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 1 -- --> v1:172.21.15.138:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11f80a1500 con 0x7f11f80a1b20 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.474 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 1 -- --> v2:172.21.15.138:3300/0 -- mon_getmap magic: 0 -- 0x7f11f809f5b0 con 0x7f11f809e500 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f11f809f850 con 0x7f11f809ee20 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient(hunting): _renew_subs 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11ff5e8640 10 monclient(hunting): authenticate will time out at 2105.913162s 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 1 --2- >> v2:172.21.15.138:3300/0 conn(0x7f11f809e500 0x7f11f809e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809ee20 0x7f11f8135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fcb83640 1 --1- >> v1:172.21.15.138:6789/0 conn(0x7f11f80a1b20 0x7f11f81357d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.138:6789/0 says I am v1:172.21.15.138:56144/0 (socket says 172.21.15.138:56144) 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient(hunting): get_auth_request con 0x7f11f809e500 auth_method 0 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fcb83640 1 -- 172.21.15.138:0/1777929671 learned_addr learned my addr 172.21.15.138:0/1777929671 (peer_addr_for_me v1:172.21.15.138:0/0) 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 10 monclient(hunting): get_auth_request con 0x7f11f809ee20 auth_method 0 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:27:02.475 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient(hunting): handle_auth_done global_id 114414 payload 274 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 1 -- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809ee20 msgr2=0x7f11f8135ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 1 --2- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809ee20 0x7f11f8135ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 1 -- 172.21.15.138:0/1777929671 >> v1:172.21.15.138:6789/0 conn(0x7f11f80a1b20 legacy=0x7f11f81357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient: _finish_hunting 0 2024-09-11T05:27:02.476 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 1 monclient: found mon.noname-a 2024-09-11T05:27:02.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:27:02.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.138:3300/0 2024-09-11T05:27:02.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fd384640 1 -- 172.21.15.138:0/1777929671 --> v2:172.21.15.138:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f809f290 con 0x7f11f809e500 2024-09-11T05:27:02.477 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.474+0000 7f11fdb85640 1 --2- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809ee20 0x7f11f8135ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:27:02.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11fd384640 1 -- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f809e500 msgr2=0x7f11f809e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:27:02.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11fd384640 1 --2- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f809e500 0x7f11f809e8e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f11f40097c0 tx=0x7f11f401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5479368271f1b21 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:27:02.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 1 -- 172.21.15.138:0/1777929671 <== mon.0 v2:172.21.15.138:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40b0070 con 0x7f11f809e500 2024-09-11T05:27:02.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:27:02.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T05:27:02.480 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: dump: 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: _finish_auth 0 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: _check_auth_tickets 2024-09-11T05:27:02.481 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 1 -- 172.21.15.138:0/1777929671 <== mon.0 v2:172.21.15.138:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f40ab600 con 0x7f11f809e500 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: handle_config config(0 keys) 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 1 -- 172.21.15.138:0/1777929671 <== mon.0 v2:172.21.15.138:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40ab9d0 con 0x7f11f809e500 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 5 monclient: authenticate success, global_id 114414 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 10 monclient: get_monmap_and_config success 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 4 set_mon_vals no callback set 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11effff640 10 monclient: dump: 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.482 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 10 monclient: shutdown 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 -- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f809e500 msgr2=0x7f11f809e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f809e500 0x7f11f809e8e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f11f40097c0 tx=0x7f11f401c360 comp rx=0 tx=0).stop 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 -- 172.21.15.138:0/1777929671 shutdown_connections 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809ee20 0x7f11f8135ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.483 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/1777929671 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f809e500 0x7f11f809e8e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 -- 172.21.15.138:0/1777929671 >> 172.21.15.138:0/1777929671 conn(0x7f11f80a3b00 msgr2=0x7f11f80a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 -- 172.21.15.138:0/1777929671 shutdown_connections 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 -- 172.21.15.138:0/1777929671 wait complete. 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 10 monclient: build_initial_monmap 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 10 monclient: monmap: 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 0 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 0.000000 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:created 0.000000 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 0 (unknown) 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.noname-a 2024-09-11T05:27:02.484 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-b 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.noname-c 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 librados: starting msgr at 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 librados: starting objecter 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 Processor -- start 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 -- start start 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 librados: setting wanted keys 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 1 librados: calling monclient init 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.478+0000 7f11ff5e8640 10 monclient: init 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient: _reopen_session rank -1 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809e500 0x7f11f811f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient(hunting): picked mon.noname-c con 0x7f11f809e500 addr [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] 2024-09-11T05:27:02.485 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11f809ee20 0x7f11f811db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient(hunting): picked mon.noname-b con 0x7f11f809ee20 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f80a1b20 0x7f11f8135580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient(hunting): picked mon.noname-a con 0x7f11f80a1b20 addr [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 -- --> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] -- mon_getmap magic: 0 -- 0x7f11f8140250 con 0x7f11f80a1b20 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f11f8136630 con 0x7f11f809ee20 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient(hunting): start opening mon connection 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 -- --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_getmap magic: 0 -- 0x7f11f81403d0 con 0x7f11f809e500 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient(hunting): _renew_subs 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient(hunting): authenticate will time out at 2105.920326s 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809e500 0x7f11f811f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 1 --2- >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f80a1b20 0x7f11f8135580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.486 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 --2- >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809e500 0x7f11f811f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.138:3301/0 says I am v2:172.21.15.138:58222/0 (socket says 172.21.15.138:58222) 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 -- 172.21.15.138:0/730168501 learned_addr learned my addr 172.21.15.138:0/730168501 (peer_addr_for_me v2:172.21.15.138:0/0) 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient(hunting): get_auth_request con 0x7f11f809e500 auth_method 0 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 10 monclient(hunting): get_auth_request con 0x7f11f80a1b20 auth_method 0 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 10 monclient(hunting): _init_auth method 2 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T05:27:02.487 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient(hunting): handle_auth_done global_id 84713 payload 824 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fcb83640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11f809ee20 0x7f11f811db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 -- 172.21.15.138:0/730168501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11f809ee20 msgr2=0x7f11f811db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11f809ee20 0x7f11f811db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 -- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f80a1b20 msgr2=0x7f11f8135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f80a1b20 0x7f11f8135580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient: _finish_hunting 0 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 monclient: found mon.noname-c 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 -- 172.21.15.138:0/730168501 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f80a6610 con 0x7f11f809e500 2024-09-11T05:27:02.488 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fdb85640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f80a1b20 0x7f11f8135580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T05:27:02.490 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 -- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809e500 msgr2=0x7f11f811f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11fd384640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809e500 0x7f11f811f720 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f11f40ae040 tx=0x7f11f4019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 client.?.objecter ms_handle_connect 0x7f11f809e500 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 client.?.objecter resend_mon_ops 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 1 -- 172.21.15.138:0/730168501 <== mon.2 v2:172.21.15.138:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40b0040 con 0x7f11f809e500 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: dump: 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.491 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: _finish_auth 0 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: _check_auth_tickets 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 5 monclient: authenticate success, global_id 84713 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient: _renew_subs 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 1 -- 172.21.15.138:0/730168501 <== mon.2 v2:172.21.15.138:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f40b9070 con 0x7f11f809e500 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.492 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f8073700 con 0x7f11f809e500 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: handle_config config(0 keys) 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 1 -- 172.21.15.138:0/730168501 <== mon.2 v2:172.21.15.138:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f40aaa10 con 0x7f11f809e500 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11edffb640 10 monclient: dump: 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:epoch 1 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:fsid 66c6f018-f82d-4af3-8f0d-6530cdae2ec3 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:last_changed 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:created 2024-09-11T05:06:55.201195+0000 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:min_mon_release 19 (squid) 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:election_strategy: 1 2024-09-11T05:27:02.493 INFO:teuthology.orchestra.run.smithi138.stderr:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2: [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] mon.c 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ef7fe640 4 set_mon_vals no callback set 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 client.84713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient: _renew_subs 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f8120f20 con 0x7f11f809e500 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.482+0000 7f11ff5e8640 1 librados: init done 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11ff5e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11ff5e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11ff5e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.494 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f809e8e0 con 0x7f11f809e500 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11edffb640 1 -- 172.21.15.138:0/730168501 <== mon.2 v2:172.21.15.138:3301/0 4 ==== mgrmap(e 5) ==== 48491+0+0 (secure 0 0 0) 0x7f11f40b3070 con 0x7f11f809e500 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11edffb640 1 --2- 172.21.15.138:0/730168501 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11dc03ced0 0x7f11dc03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11edffb640 1 -- 172.21.15.138:0/730168501 <== mon.2 v2:172.21.15.138:3301/0 5 ==== osd_map(698..698 src has 520..698) ==== 4717+0+0 (secure 0 0 0) 0x7f11f40e91d0 con 0x7f11f809e500 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11edffb640 10 client.84713.objecter ms_dispatch 0x7f11f8000ba0 osd_map(698..698 src has 520..698) 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11edffb640 3 client.84713.objecter handle_osd_map got epochs [698,698] > 0 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11edffb640 3 client.84713.objecter handle_osd_map decoding full epoch 698 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11edffb640 20 client.84713.objecter dump_active .. 0 homeless 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11fcb83640 1 --2- 172.21.15.138:0/730168501 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11dc03ced0 0x7f11dc03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11fcb83640 10 monclient: get_auth_request con 0x7f11dc03ced0 auth_method 0 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11edffb640 10 client.84713.objecter ms_handle_connect 0x7f11dc03ced0 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.486+0000 7f11fcb83640 1 --2- 172.21.15.138:0/730168501 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11dc03ced0 0x7f11dc03f390 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f11f813d630 tx=0x7f11e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.490+0000 7f11edffb640 1 -- 172.21.15.138:0/730168501 <== mon.2 v2:172.21.15.138:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f11f80a6610 con 0x7f11f809e500 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.490+0000 7f11edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T05:27:02.495 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.490+0000 7f11edffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T05:27:02.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.658+0000 7f11ff5e8640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T05:27:02.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.658+0000 7f11ff5e8640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T05:27:02.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.658+0000 7f11ff5e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.138:3301/0 2024-09-11T05:27:02.657 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.658+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 --> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f11f8120aa0 con 0x7f11f809e500 2024-09-11T05:27:02.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.666+0000 7f11edffb640 1 -- 172.21.15.138:0/730168501 <== mon.2 v2:172.21.15.138:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f11f40b4e20 con 0x7f11f809e500 2024-09-11T05:27:02.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.666+0000 7f11edffb640 10 monclient: handle_config config(1 keys) 2024-09-11T05:27:02.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.666+0000 7f11ef7fe640 4 set_mon_vals no callback set 2024-09-11T05:27:02.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.666+0000 7f11ef7fe640 10 set_mon_vals mon_health_to_clog = false 2024-09-11T05:27:02.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.666+0000 7f11edffb640 1 -- 172.21.15.138:0/730168501 <== mon.2 v2:172.21.15.138:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7f11f8120aa0 con 0x7f11f809e500 2024-09-11T05:27:02.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.666+0000 7f11edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T05:27:02.664 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.666+0000 7f11edffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 10 librados: watch_flush enter 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 10 librados: watch_flush exit 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 20 client.84713.objecter shutdown clearing up homeless session... 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 10 client.84713.objecter successfully canceled tick 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11dc03ced0 msgr2=0x7f11dc03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/730168501 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11dc03ced0 0x7f11dc03f390 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f11f813d630 tx=0x7f11e8009040 comp rx=0 tx=0).stop 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 10 monclient: shutdown 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809e500 msgr2=0x7f11f811f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T05:27:02.668 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809e500 0x7f11f811f720 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f11f40ae040 tx=0x7f11f4019100 comp rx=0 tx=0).stop 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 shutdown_connections 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/730168501 >> v2:172.21.15.190:6800/1562782189 conn(0x7f11dc03ced0 0x7f11dc03f390 unknown :-1 s=CLOSED pgs=3537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f11f80a1b20 0x7f11f8135580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11f809ee20 0x7f11f811db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 --2- 172.21.15.138:0/730168501 >> [v2:172.21.15.138:3301/0,v1:172.21.15.138:6790/0] conn(0x7f11f809e500 0x7f11f811f720 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 >> 172.21.15.138:0/730168501 conn(0x7f11f80a3b00 msgr2=0x7f11f80a2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 shutdown_connections 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 -- 172.21.15.138:0/730168501 wait complete. 2024-09-11T05:27:02.669 INFO:teuthology.orchestra.run.smithi138.stderr:2024-09-11T05:27:02.670+0000 7f11ff5e8640 1 librados: shutdown 2024-09-11T05:27:02.685 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T05:27:02.685 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T05:27:02.685 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T05:27:02.685 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:02.759 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T05:27:02.759 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T05:27:02.759 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:02.855 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T05:27:02.855 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T05:27:02.855 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:02.933 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T05:27:02.933 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T05:27:02.933 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:03.024 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T05:27:03.024 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-11T05:27:03.024 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:06.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-11T05:27:09.125 INFO:tasks.ceph.osd.4:Stopped 2024-09-11T05:27:09.126 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-11T05:27:09.126 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:09.194 INFO:tasks.ceph.osd.5:Stopped 2024-09-11T05:27:09.194 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T05:27:09.194 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T05:27:09.194 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:09.225 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T05:27:09.225 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T05:27:09.225 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T05:27:09.225 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:09.253 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T05:27:09.254 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T05:27:09.254 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:09.323 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T05:27:09.323 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T05:27:09.324 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T05:27:09.345 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T05:27:09.345 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T05:27:09.345 DEBUG:teuthology.orchestra.run.smithi138:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-11T05:27:09.386 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T05:27:09.386 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T05:27:09.704 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T05:27:09.704 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T05:27:09.914 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T05:27:09.914 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T05:27:10.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T05:27:10.079 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T05:27:10.343 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T05:27:10.344 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-11T05:27:10.470 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T05:27:10.471 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-11T05:27:10.607 INFO:tasks.ceph:Archiving mon data... 2024-09-11T05:27:10.608 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/data/mon.a.tgz 2024-09-11T05:27:10.609 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-09-11T05:27:10.614 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.zM8hdgdpbe 2024-09-11T05:27:10.615 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zM8hdgdpbe 2024-09-11T05:27:10.781 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.zM8hdgdpbe 2024-09-11T05:27:10.843 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.zM8hdgdpbe is 608KB 2024-09-11T05:27:10.905 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.zM8hdgdpbe 2024-09-11T05:27:10.910 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/data/mon.c.tgz 2024-09-11T05:27:10.910 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-09-11T05:27:10.959 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.Kgy9CGTn1p 2024-09-11T05:27:10.959 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Kgy9CGTn1p 2024-09-11T05:27:11.123 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.Kgy9CGTn1p 2024-09-11T05:27:11.187 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.Kgy9CGTn1p is 614KB 2024-09-11T05:27:11.249 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.Kgy9CGTn1p 2024-09-11T05:27:11.255 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/data/mon.b.tgz 2024-09-11T05:27:11.255 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-11T05:27:11.261 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.fL9C6xDxt7 2024-09-11T05:27:11.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.fL9C6xDxt7 2024-09-11T05:27:11.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.fL9C6xDxt7 2024-09-11T05:27:11.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-11T05:27:11.492 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.fL9C6xDxt7 is 604KB 2024-09-11T05:27:11.554 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.fL9C6xDxt7 2024-09-11T05:27:11.559 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T05:27:11.560 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T05:27:11.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T05:27:11.622 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T05:27:11.668 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T05:27:11.669 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T05:27:11.671 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/remote/smithi138/crash 2024-09-11T05:27:11.671 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T05:27:11.684 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/remote/smithi190/crash 2024-09-11T05:27:11.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T05:27:11.724 INFO:tasks.ceph:Compressing logs... 2024-09-11T05:27:11.725 DEBUG:teuthology.orchestra.run.smithi138:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T05:27:11.728 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T05:27:11.751 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30194.log 2024-09-11T05:27:11.752 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29650.log 2024-09-11T05:27:11.752 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33061.log 2024-09-11T05:27:11.753 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30194.log: /var/log/ceph/ceph-client.admin.29650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30220.log 2024-09-11T05:27:11.753 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33529.log 2024-09-11T05:27:11.753 INFO:teuthology.orchestra.run.smithi138.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29650.log.gz 2024-09-11T05:27:11.754 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30220.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33061.log.gz 2024-09-11T05:27:11.754 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33243.log 2024-09-11T05:27:11.755 INFO:teuthology.orchestra.run.smithi138.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30194.log.gz 2024-09-11T05:27:11.755 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28124.log 2024-09-11T05:27:11.755 INFO:teuthology.orchestra.run.smithi138.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30220.log.gz 2024-09-11T05:27:11.755 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32280.log 2024-09-11T05:27:11.755 INFO:teuthology.orchestra.run.smithi138.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33529.log.gz 2024-09-11T05:27:11.755 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33477.log 87.4% 2024-09-11T05:27:11.755 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33243.log.gz 2024-09-11T05:27:11.755 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29918.log 2024-09-11T05:27:11.759 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33477.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28124.log.gz 2024-09-11T05:27:11.759 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29572.log 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29918.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33477.log.gz 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32280.log.gzgzip 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33683.log 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29918.log.gz 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.29572.log: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28130.log 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.0.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29572.log.gz 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30337.log 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33683.log.gz 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28717.log 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30337.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.28130.log.gz 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28015.log 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32821.log 2024-09-11T05:27:11.760 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28015.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30337.log.gz 2024-09-11T05:27:11.761 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32359.log 2024-09-11T05:27:11.761 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28717.log.gz 2024-09-11T05:27:11.761 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33347.log 2024-09-11T05:27:11.761 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28015.log.gz 2024-09-11T05:27:11.761 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30022.log 2024-09-11T05:27:11.761 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32821.log.gz 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28639.log 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30022.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32359.log.gz 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28795.log 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30022.log.gz 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33347.log.gz 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31630.log 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33005.log 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28795.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28639.log.gz 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30469.log 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28795.log.gz 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29235.log 2024-09-11T05:27:11.767 INFO:teuthology.orchestra.run.smithi138.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31630.log.gz 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31317.log 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33005.log.gz 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30923.log 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30469.log.gz 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31395.log 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29235.log.gz 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31317.log.gz 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31918.log 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32200.log 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30923.log.gz 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31918.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31395.log.gz 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29291.log 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32082.log 2024-09-11T05:27:11.768 INFO:teuthology.orchestra.run.smithi138.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31918.log.gz 2024-09-11T05:27:11.769 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31027.log 2024-09-11T05:27:11.769 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32200.log.gz 2024-09-11T05:27:11.769 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27970.log 2024-09-11T05:27:11.769 INFO:teuthology.orchestra.run.smithi138.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29291.log.gz 2024-09-11T05:27:11.769 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29507.log 2024-09-11T05:27:11.769 INFO:teuthology.orchestra.run.smithi138.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32082.log.gz 2024-09-11T05:27:11.769 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.27970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31814.log 2024-09-11T05:27:11.779 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29507.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31027.log.gz 2024-09-11T05:27:11.779 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30415.log 2024-09-11T05:27:11.779 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27970.log.gz 2024-09-11T05:27:11.779 INFO:teuthology.orchestra.run.smithi138.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29507.log.gz 2024-09-11T05:27:11.779 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28821.log 2024-09-11T05:27:11.779 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28067.log 2024-09-11T05:27:11.779 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30415.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31814.log.gz 2024-09-11T05:27:11.779 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31710.log 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30415.log.gz 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31866.log 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28821.log.gz 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30248.log 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28067.log.gz 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31866.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21480.log 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30248.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31710.log.gz 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29861.log 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31866.log.gz 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30248.log.gz 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph.tmp-client.admin.21480.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.31551.log 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.21480.log.gz 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29702.log 2024-09-11T05:27:11.780 INFO:teuthology.orchestra.run.smithi138.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29861.log.gz 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29265.log 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32847.log 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29265.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31551.log.gz 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29129.log 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29265.log.gz 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29702.log.gz 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28743.log 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29129.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32847.log.gz 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29944.log 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29129.log.gz 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31001.log 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30521.log 2024-09-11T05:27:11.781 INFO:teuthology.orchestra.run.smithi138.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28743.log.gz 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28132.log 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29944.log.gz 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29833.log 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31001.log.gz 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29970.log 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30521.log.gz 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33607.log 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28132.log.gz 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32228.log 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29833.log.gz 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28899.log 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32228.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29970.log.gz 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T05:27:11.782 INFO:teuthology.orchestra.run.smithi138.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33607.log.gz 2024-09-11T05:27:11.783 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31577.log 2024-09-11T05:27:11.783 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32228.log.gz 2024-09-11T05:27:11.783 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32661.log 2024-09-11T05:27:11.783 INFO:teuthology.orchestra.run.smithi138.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28899.log.gz 2024-09-11T05:27:11.783 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29598.log 2024-09-11T05:27:11.789 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32979.log 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31577.log.gz 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27945.log 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29598.log: /var/log/ceph/ceph-client.admin.32979.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32661.log.gz 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29317.log 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32489.log 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.27945.log: -- replaced with /var/log/ceph/ceph-client.admin.29598.log.gz 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30363.log 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.32979.log.gz 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27945.log.gz 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32489.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29317.log.gz 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32489.log.gz 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29676.log 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31187.log 2024-09-11T05:27:11.790 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33191.log 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30363.log.gz 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28136.log 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29676.log.gz 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28977.log 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31187.log.gz 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32411.log 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33191.log.gz 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29059.log 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28136.log.gz 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32901.log 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.28977.log.gz 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28493.log 2024-09-11T05:27:11.791 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32411.log.gz 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32463.log 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29059.log.gz 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31079.log 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32901.log.gz 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28313.log 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28493.log.gz 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33373.log 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32463.log.gz 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28041.log 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32633.log 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31079.log.gz 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28313.log.gz 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30870.log 2024-09-11T05:27:11.792 INFO:teuthology.orchestra.run.smithi138.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33373.log.gz 2024-09-11T05:27:11.793 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32633.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28041.log.gz 2024-09-11T05:27:11.793 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32567.log 2024-09-11T05:27:11.793 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32741.log 2024-09-11T05:27:11.793 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30870.log: /var/log/ceph/ceph-client.admin.32567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28873.log 2024-09-11T05:27:11.793 INFO:teuthology.orchestra.run.smithi138.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32633.log.gz 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29863.log 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29395.log 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28636.log 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28081.log 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29023.log 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25211.log 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28235.log 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25211.log: gzip -5 --verbose 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29863.log.gz 2024-09-11T05:27:11.818 INFO:teuthology.orchestra.run.smithi190.stderr: -- /var/log/ceph/ceph-client.admin.28094.log 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28235.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.25211.log.gz 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28672.log 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29395.log.gz 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28245.log 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28636.log.gz 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28735.log 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29647.log 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28089.log 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29647.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28735.log.gz 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29864.log 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28089.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.28094.log.gz 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28516.log 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29864.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.29647.log.gz 2024-09-11T05:27:11.819 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28439.log 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28516.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.28672.log.gz 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28041.log 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28235.log.gz 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.28089.log.gz/var/log/ceph/ceph-client.admin.28439.log: 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29328.log 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29864.log.gz/var/log/ceph/ceph-client.admin.28041.log: 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25163.log 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29394.log 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25163.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25163.log.gz 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28245.log.gz 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29736.log 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29394.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr: 91.5% 90.4% -- replaced with /var/log/ceph/ceph-client.admin.29023.log.gz 2024-09-11T05:27:11.820 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29328.log.gz/var/log/ceph/ceph-client.admin.29736.log: 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28145.log 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.3.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29736.log.gz 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29737.log 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0%/var/log/ceph/ceph-osd.5.log: -- replaced with /var/log/ceph/ceph-client.admin.29394.log.gz 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29644.log 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29737.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.28439.log.gz 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29030.log 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29644.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29737.log.gz 2024-09-11T05:27:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29048.log 2024-09-11T05:27:11.827 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29030.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29644.log.gz 2024-09-11T05:27:11.827 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25139.log 2024-09-11T05:27:11.827 INFO:teuthology.orchestra.run.smithi190.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29030.log.gz 2024-09-11T05:27:11.827 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28311.log 2024-09-11T05:27:11.827 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25139.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.25139.log.gz 2024-09-11T05:27:11.827 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28187.log 2024-09-11T05:27:11.827 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28311.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28311.log.gz 2024-09-11T05:27:11.827 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29423.log 2024-09-11T05:27:11.830 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28847.log 2024-09-11T05:27:11.830 INFO:teuthology.orchestra.run.smithi138.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30870.log.gz 2024-09-11T05:27:11.830 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32567.log.gz 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29449.log 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28873.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32741.log.gz 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30677.log 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29449.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32875.log 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28873.log.gz 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28847.log.gz 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31762.log 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29449.log.gz 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33659.log 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29996.log 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30677.log.gz 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32875.log.gz 2024-09-11T05:27:11.831 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30074.log 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31762.log.gz 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30547.log 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33659.log.gz 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31473.log 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30547.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29996.log.gz 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28587.log 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30074.log.gz 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33031.log 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30547.log.gz 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33217.log 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31473.log.gz 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28487.log 2024-09-11T05:27:11.832 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28587.log.gz 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30128.log 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33031.log.gz 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29728.log 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33217.log.gz 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28139.log 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29728.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28487.log.gz 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31265.log 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30128.log.gz 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29371.log 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29728.log.gz 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28431.log 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28139.log.gz 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31499.log 2024-09-11T05:27:11.833 INFO:teuthology.orchestra.run.smithi138.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31265.log.gz 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33451.log 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29371.log.gz 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31499.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28431.log.gz 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30949.log 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31499.log.gz 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29157.log 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33451.log.gz 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30599.log 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29397.log 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30949.log.gz 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31736.log 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29157.log.gz/var/log/ceph/ceph-client.admin.29397.log: 2024-09-11T05:27:11.834 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29003.log 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30599.log.gz 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31736.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29397.log.gz 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31736.log.gz 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32687.log 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30723.log 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28311.log 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32332.log 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29003.log.gz 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32687.log.gz 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28311.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30723.log.gz 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28311.log.gz 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28435.log 2024-09-11T05:27:11.835 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30443.log 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33555.log 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32332.log.gz/var/log/ceph/ceph-client.admin.30443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33087.log 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28435.log.gz 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33555.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30443.log.gz 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.33555.log.gz 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27995.log 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33113.log 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.27995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28093.log 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33087.log.gz 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32515.log 2024-09-11T05:27:11.836 INFO:teuthology.orchestra.run.smithi138.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27995.log.gz -5 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29423.log 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28093.log: /var/log/ceph/ceph-client.admin.32515.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33113.log.gz 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28093.log.gz 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32515.log.gz 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30102.log 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28613.log 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31656.log 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30651.log 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.29423.log.gz 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30102.log.gz 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31656.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28613.log.gz 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31656.log.gz 2024-09-11T05:27:11.837 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29209.log 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29781.log 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29209.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30651.log.gz 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27872.log 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr: 89.1%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.29209.log.gz 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32048.log 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29781.log.gz 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.27872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30844.log 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28309.log 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27872.log.gz 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30844.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32048.log.gz 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31133.log 2024-09-11T05:27:11.838 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30844.log.gz 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29475.log 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29624.log 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28309.log.gz 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33581.log 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31133.log.gz 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29624.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29475.log.gz 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29624.log.gz 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29892.log 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29183.log 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31892.log 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33581.log.gz 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29183.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29892.log.gz 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29183.log.gz 2024-09-11T05:27:11.839 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32147.log 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31239.log 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31291.log 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31892.log.gz/var/log/ceph/ceph-client.admin.31239.log: 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32147.log.gz 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31239.log.gz 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28769.log 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28275.log 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32254.log 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31291.log.gz 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28275.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28769.log.gz 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28275.log.gz 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30495.log 2024-09-11T05:27:11.840 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32254.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-11T05:27:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30816.log 2024-09-11T05:27:11.841 INFO:teuthology.orchestra.run.smithi138.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32254.log.gz 2024-09-11T05:27:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-mon.c.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30495.log.gz 2024-09-11T05:27:11.841 INFO:teuthology.orchestra.run.smithi138.stderr: 91.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T05:27:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30276.log 2024-09-11T05:27:11.841 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32437.log 2024-09-11T05:27:11.842 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28187.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T05:27:11.842 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29423.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29423.log.gz 2024-09-11T05:27:11.847 INFO:teuthology.orchestra.run.smithi190.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28041.log.gz 2024-09-11T05:27:11.847 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28974.log 2024-09-11T05:27:11.857 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32953.log 2024-09-11T05:27:11.857 INFO:teuthology.orchestra.run.smithi138.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30816.log.gz 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32437.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30276.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31421.log 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31213.log 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32437.log.gz 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31421.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32953.log.gz 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30311.log 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31421.log.gz 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28138.log 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30311.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31213.log.gz 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32927.log 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30311.log.gz 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph.audit.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33139.log 2024-09-11T05:27:11.858 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28138.log.gz 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28665.log 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33139.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32927.log.gz 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33139.log.gz 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33165.log 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29343.log 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33165.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28665.log.gz 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33321.log 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33165.log.gz 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30975.log 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33321.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29343.log.gz 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33425.log 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr: 87.9%/var/log/ceph/ceph-client.admin.30975.log: -- replaced with /var/log/ceph/ceph-client.admin.33321.log.gz 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32306.log 2024-09-11T05:27:11.859 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30975.log.gz 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31788.log 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32306.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33425.log.gz 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32306.log.gz 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25274.log 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30896.log 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31788.log: /var/log/ceph/ceph-client.admin.25274.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31788.log.gz 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25274.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32385.log 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32174.log 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30896.log: /var/log/ceph/ceph-client.admin.32385.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.30896.log.gz 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32385.log.gz 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32022.log 2024-09-11T05:27:11.860 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32795.log 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32174.log: /var/log/ceph/ceph-client.admin.32022.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32174.log.gz 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31343.log 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32022.log.gz 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32795.log: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32795.log.gz 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33399.log 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28457.log 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31944.log 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31343.log.gz 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28457.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33399.log.gz 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28457.log.gz 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31369.log 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31996.log 2024-09-11T05:27:11.861 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31944.log: /var/log/ceph/ceph-client.admin.31369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31840.log 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31944.log.gz 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31996.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31369.log.gz 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.31996.log.gz 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31107.log 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31525.log 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33503.log 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31107.log: /var/log/ceph/ceph-client.admin.31525.log: 89.0% 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31840.log.gz 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31107.log.gz 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28464.log 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.31525.log.gz 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28925.log 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33503.log: /var/log/ceph/ceph-client.admin.28464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30048.log 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28925.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33503.log.gz 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28464.log.gz 2024-09-11T05:27:11.862 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32593.log 2024-09-11T05:27:11.863 INFO:teuthology.orchestra.run.smithi138.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28925.log.gz 2024-09-11T05:27:11.863 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30573.log 2024-09-11T05:27:11.863 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30048.log: /var/log/ceph/ceph-client.admin.32593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32541.log 2024-09-11T05:27:11.867 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mgr.x.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.28516.log.gz 2024-09-11T05:27:11.867 INFO:teuthology.orchestra.run.smithi190.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.28081.log.gz 2024-09-11T05:27:11.868 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29329.log 2024-09-11T05:27:11.868 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28685.log 2024-09-11T05:27:11.868 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28974.log: /var/log/ceph/ceph-client.admin.29329.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29329.log.gz 2024-09-11T05:27:11.868 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T05:27:11.868 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28685.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28685.log.gz 2024-09-11T05:27:11.868 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28940.log 2024-09-11T05:27:11.868 INFO:teuthology.orchestra.run.smithi190.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.29048.log.gz 2024-09-11T05:27:11.868 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28683.log 2024-09-11T05:27:11.874 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30573.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30048.log.gz 2024-09-11T05:27:11.874 INFO:teuthology.orchestra.run.smithi138.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32593.log.gz 2024-09-11T05:27:11.874 INFO:teuthology.orchestra.run.smithi138.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30573.log.gzgzip 2024-09-11T05:27:11.874 INFO:teuthology.orchestra.run.smithi138.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29755.log 2024-09-11T05:27:11.874 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31684.log 2024-09-11T05:27:11.874 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32541.log: gzip/var/log/ceph/ceph-client.admin.29755.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30389.log 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31684.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29755.log.gz 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.32541.log.gz 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31684.log.gz 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28279.log 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33633.log 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33295.log 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30389.log.gz 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33633.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28279.log.gz 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33633.log.gz 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31970.log 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31447.log 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33295.log: /var/log/ceph/ceph-client.admin.31970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29807.log 2024-09-11T05:27:11.875 INFO:teuthology.orchestra.run.smithi138.stderr: 88.8%/var/log/ceph/ceph-client.admin.31447.log: -- replaced with /var/log/ceph/ceph-client.admin.33295.log.gz 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31970.log.gz 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31447.log.gz 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31053.log 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30625.log 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.31053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25197.log 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29807.log.gz 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.30625.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31053.log.gz 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30625.log.gz 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30788.log 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32715.log 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.25197.log: /var/log/ceph/ceph-client.admin.30788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32769.log 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32715.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25197.log.gz 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30788.log.gz 2024-09-11T05:27:11.876 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28312.log 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32715.log.gz 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.32769.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32769.log.gz 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25080.log 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28951.log 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28312.log: /var/log/ceph/ceph-client.admin.25080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33269.log 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28951.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28312.log.gz 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28951.log.gz 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.25080.log.gz 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29029.log 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.33269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28691.log 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.29029.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33269.log.gz 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.28691.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29029.log.gz 2024-09-11T05:27:11.877 INFO:teuthology.orchestra.run.smithi138.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28691.log.gz 2024-09-11T05:27:11.881 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28940.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28940.log.gz 2024-09-11T05:27:11.881 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29422.log 2024-09-11T05:27:11.881 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28683.log: 91.0% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28974.log.gz 2024-09-11T05:27:11.881 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28683.log.gz 2024-09-11T05:27:11.881 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28057.log 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29327.log 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28057.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28479.log 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29327.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29422.log.gz 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29327.log.gz 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28606.log 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr: 91.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28166.log 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28057.log.gz 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28479.log: /var/log/ceph/ceph-client.admin.28606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25235.log 2024-09-11T05:27:11.882 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28166.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28606.log.gz 2024-09-11T05:27:11.883 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28934.log 2024-09-11T05:27:11.883 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25235.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.25235.log.gz 2024-09-11T05:27:11.883 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28073.log 2024-09-11T05:27:11.891 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28934.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28479.log.gz 2024-09-11T05:27:11.891 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28915.log 2024-09-11T05:27:11.893 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28073.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.28073.log.gz 2024-09-11T05:27:11.893 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.28934.log.gz 2024-09-11T05:27:11.894 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T05:27:11.895 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29645.log 2024-09-11T05:27:11.902 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.audit.log: 92.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T05:27:11.903 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28868.log 2024-09-11T05:27:11.905 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29645.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29645.log.gz 2024-09-11T05:27:11.906 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28280.log 2024-09-11T05:27:11.919 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28868.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.28915.log.gz 2024-09-11T05:27:11.920 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29648.log 2024-09-11T05:27:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28280.log: 91.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25187.log 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29648.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.28868.log.gz 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29648.log.gz 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28280.log.gz 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28662.log 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29396.log 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28662.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.25187.log.gz 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28662.log.gz 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28356.log 2024-09-11T05:27:11.938 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28332.log 2024-09-11T05:27:11.940 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29396.log: /var/log/ceph/ceph-client.admin.28356.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29396.log.gz 2024-09-11T05:27:11.941 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28789.log 2024-09-11T05:27:11.945 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28332.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.28356.log.gz 2024-09-11T05:27:11.947 INFO:teuthology.orchestra.run.smithi190.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28332.log.gz 2024-09-11T05:27:11.947 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25266.log 2024-09-11T05:27:11.948 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28049.log 2024-09-11T05:27:11.948 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28789.log: /var/log/ceph/ceph-client.admin.25266.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.25266.log.gz 2024-09-11T05:27:11.956 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-11T05:27:11.956 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28638.log 2024-09-11T05:27:11.964 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28049.log: /var/log/ceph/ceph-mon.b.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.28166.log.gz 2024-09-11T05:27:11.967 INFO:teuthology.orchestra.run.smithi190.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.28789.log.gz 2024-09-11T05:27:11.967 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-11T05:27:11.970 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28638.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28638.log.gz 2024-09-11T05:27:11.970 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28810.log 2024-09-11T05:27:11.984 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28065.log 2024-09-11T05:27:11.986 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28810.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28810.log.gz 2024-09-11T05:27:11.988 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28522.log 2024-09-11T05:27:12.004 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28065.log: 93.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29646.log 2024-09-11T05:27:12.010 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28522.log: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.28522.log.gz 2024-09-11T05:27:12.020 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28661.log 2024-09-11T05:27:12.023 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29646.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29646.log.gz 2024-09-11T05:27:12.029 INFO:teuthology.orchestra.run.smithi190.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28065.log.gz 2024-09-11T05:27:12.029 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28049.log.gz 2024-09-11T05:27:12.032 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29735.log 2024-09-11T05:27:12.035 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28661.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28661.log.gz 2024-09-11T05:27:12.035 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28303.log 2024-09-11T05:27:12.039 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29735.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29735.log.gz 2024-09-11T05:27:12.039 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28444.log 2024-09-11T05:27:12.043 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28303.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.28303.log.gz 2024-09-11T05:27:12.043 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29862.log 2024-09-11T05:27:12.049 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28444.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.28444.log.gz 2024-09-11T05:27:12.049 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28507.log 2024-09-11T05:27:12.053 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29862.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29862.log.gz 2024-09-11T05:27:12.053 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28642.log 2024-09-11T05:27:12.056 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28507.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28507.log.gz 2024-09-11T05:27:12.057 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28642.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28642.log.gz 2024-09-11T05:27:12.286 INFO:teuthology.orchestra.run.smithi190.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T05:27:12.333 INFO:teuthology.orchestra.run.smithi190.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.28187.log.gz 2024-09-11T05:27:12.890 INFO:teuthology.orchestra.run.smithi138.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-11T05:27:12.893 INFO:teuthology.orchestra.run.smithi138.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T05:27:13.068 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28145.log.gz 2024-09-11T05:27:13.173 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-11T05:27:18.224 INFO:teuthology.orchestra.run.smithi138.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T05:27:18.428 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-11T05:27:18.782 INFO:teuthology.orchestra.run.smithi138.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T05:27:18.800 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T05:27:19.288 INFO:teuthology.orchestra.run.smithi138.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T05:27:19.289 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-11T05:27:19.289 INFO:teuthology.orchestra.run.smithi138.stderr:real 0m7.559s 2024-09-11T05:27:19.289 INFO:teuthology.orchestra.run.smithi138.stderr:user 0m22.667s 2024-09-11T05:27:19.289 INFO:teuthology.orchestra.run.smithi138.stderr:sys 0m0.850s 2024-09-11T05:27:19.884 INFO:teuthology.orchestra.run.smithi190.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-11T05:27:19.885 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T05:27:19.885 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m8.113s 2024-09-11T05:27:19.885 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m24.749s 2024-09-11T05:27:19.886 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m0.897s 2024-09-11T05:27:19.886 INFO:tasks.ceph:Archiving logs... 2024-09-11T05:27:19.887 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/remote/smithi138/log 2024-09-11T05:27:19.888 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T05:27:23.122 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/remote/smithi190/log 2024-09-11T05:27:23.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T05:27:26.405 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T05:27:26.413 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T05:27:26.413 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T05:27:26.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T05:27:26.489 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T05:27:26.498 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T05:27:26.499 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:27:26.501 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T05:27:26.512 INFO:teuthology.orchestra.run.smithi138.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:27:26.512 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================== 2024-09-11T05:27:26.512 INFO:teuthology.orchestra.run.smithi138.stdout:*hv01.front.sepi 67.205.162.81 3 u 13 64 377 0.092 -6.991 1.157 2024-09-11T05:27:26.512 INFO:teuthology.orchestra.run.smithi138.stdout:+hv02.front.sepi 63.231.80.2 3 u 28 64 377 0.088 -3.996 0.848 2024-09-11T05:27:26.512 INFO:teuthology.orchestra.run.smithi138.stdout:+hv03.front.sepi 74.6.168.72 3 u 32 64 377 0.085 -6.294 0.689 2024-09-11T05:27:26.512 INFO:teuthology.orchestra.run.smithi138.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-11T05:27:26.514 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T05:27:26.514 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2024-09-11T05:27:26.514 INFO:teuthology.orchestra.run.smithi190.stdout:*hv01.front.sepi 67.205.162.81 3 u 26 64 377 0.089 -8.672 0.580 2024-09-11T05:27:26.514 INFO:teuthology.orchestra.run.smithi190.stdout:+hv02.front.sepi 63.231.80.2 3 u 29 64 377 0.087 -6.637 0.519 2024-09-11T05:27:26.515 INFO:teuthology.orchestra.run.smithi190.stdout:+hv03.front.sepi 74.6.168.72 3 u 32 64 377 0.084 -8.631 0.580 2024-09-11T05:27:26.515 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-11T05:27:26.515 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T05:27:26.583 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T05:27:26.584 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T05:27:26.633 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T05:27:26.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T05:27:26.731 INFO:teuthology.task.internal:Duration was 1799.716025 seconds 2024-09-11T05:27:26.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T05:27:26.781 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T05:27:26.781 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T05:27:26.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T05:27:26.822 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T05:27:26.822 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-09-11T05:27:26.823 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T05:27:26.872 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T05:27:26.872 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T05:27:26.882 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T05:27:26.882 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T05:27:26.916 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T05:27:26.950 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T05:27:26.950 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T05:27:26.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T05:27:27.049 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T05:27:27.060 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T05:27:27.060 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T05:27:27.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T05:27:27.082 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T05:27:27.099 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:27:27.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:27:27.125 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-09-11T05:27:27.132 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-09-11T05:27:27.154 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:27:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:27:27.204 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T05:27:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T05:27:27.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T05:27:27.218 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T05:27:27.219 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/remote/smithi138 2024-09-11T05:27:27.219 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T05:27:27.268 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900187/remote/smithi190 2024-09-11T05:27:27.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T05:27:27.315 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T05:27:27.316 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T05:27:27.318 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T05:27:27.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T05:27:27.345 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T05:27:27.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T05:27:27.374 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T05:27:27.374 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T05:27:27.376 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T05:27:27.380 INFO:teuthology.orchestra.run.smithi138.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 05:27 /home/ubuntu/cephtest 2024-09-11T05:27:27.382 INFO:teuthology.orchestra.run.smithi190.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 05:27 /home/ubuntu/cephtest 2024-09-11T05:27:27.384 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T05:27:27.424 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T05:27:27.528 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1799.7160246372223 flavor: default owner: scheduled_hyelloji@teuthology success: true 2024-09-11T05:27:27.529 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T05:27:27.614 INFO:teuthology.run:pass